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Abstract 

In this paper we study the monodromy action on the first Betti and de Rham non- 
abelian cohomology arising from a family of smooth curves. We describe sufficient 
conditions for the existence of a Zariski dense monodromy orbit. In particular we show 
that for a Lefschetz pencil of sufficiently high degree the monodromy action is dense. 
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1 Introduction 



We work in the category of quasi-projective schemes over C. Let / : X — > B be a smooth 
projective morphism with connected fibers of dimension one and genus at least two. Fix a 
base point o G B and let X Q be the corresponding fiber of /. In this paper we study the 
monodromy action of 7ti(B, o) on the degree one non-abelian Betti and de Rham cohomology 
of X Q . Let tti(X ) denote the abstract fundamental group of X Q and let 

mon : tti(B,o) — > Out(7Tipf )) 

be the geometric monodromy representation of the family / : X — > B. For any positive 
integer n let 

mon^ : m(B, o) Aut(#£(X , GL(n, C))) 

be the induced monodromy action on the non-abelian Betti cohomology with coefficients in 
GL(n,C). 

The de Rham object which corresponds to mon^ is the Gauss-Manin connection 
on the relative de Rham cohomology stack H}) R (X/B, GL(n, C)). While the non-abelian 
Betti and de Rham cohomology are most naturally viewed as stacks, for the purposes of 
the present paper it suffices to work with the corresponding coarse moduli spaces. To 
indicate that we will write M B (X Q ,n) and M DR (X/ B,n) rather than H B (X Q , GL(n, C)) 
and H}) R (X/B, GL(n, C)). Concretely M B (X ,n) denotes the moduli space of (semisim- 
plifications of) represenations of 7Ti(X ) in GL(n, C) and M DR (X Q ,n) denotes the moduli 
space of rank n algebraic local systems of complex vector spaces on X Q . The total space 
Md R (X/ B,n) — > B is a quasiprojective variety over B whose fiber over the point o is 
M DR (X ,n). 

For a loop 7 G iri(B,o) the action of mon^(7) on M B (X Q ,n) is given by composing a 
n-dimensional represenation of 7r 1 (X ) with some lift of the outer automorphism mon(7) of 
7Ti(X ). This gives a well defined action on conjugacy classes of representations of tti(X ) 
and results in an algebraic automorphism mon^^) : M B (X ,n) — > Mb(X ,ti). 

There is an analytic action mon^ K of 7Ti(B,o) on Mr) R (X ,n) which is most naturally 
described through the Riemann-Hilbert correspondence (see e.g. |Deligne73| , |Simpso n95 , 
Section 7]). It is shown in |Simpson95| Section 7] that the passage from a local system to 
its monodromy representation induces an isomorphism of analytic spaces 

r : M DR (X ,n)^M B (X ,nr. 

Now given 7 we can define an analytic automorphism of mon^ R (7) of Mdr(X , n) by putting 
mon2, J? (7) = r^ 1 omon'^(7) or. This analytic action is the monodromy of the algebraic non- 
abelian Gauss-Manin connection V£, R on the total space Mdr(X/ 'B,n) |Simpson95| Sec- 
tion 8]. 

It is natural to try to understand the complexity of the algebraic (respectively analytic) 
action of iri(B,o) on M B (X ,n) (respectively M DR (X ,n)) by measuring in some way the 
size of the 7Ti(B, o)-orbits on M B (X ,n) and M DR (X ,n). Analogous questions concerning 
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the monodromy action of 7Ti(B,o) on spaces of special representations of 7Ti(X ) (e.g. real 
representations, representations with compact image, projective structures, etc.) have been 
the focus of active research in the recent years |Goldman97j . |McMullen00j . |Gallo et~al .00\. 



In that direction the result most relevant to our setup is a theorem of W. Goldman who 
showed in |Goldman97j that the mapping class group acts ergodically on the space of all 
representations of 7Ti(X ) into SU(2). Unfortunately, Goldman's proof does not generalize to 
the case of representations into SU(n) for n > 2 and we do not know whether the mapping 
class group of X D still acts ergodically on the space of such representations. Instead of 
pursuing the ergodicity question in its full generality we chose to work with non-abelian 
cohomology with complex coefficients. This allows us to use the algebraic (respectively 
analytic) nature of the monodromy action on M#(X ,n) (respectively M£, R {X 0) n)) and to 
describe the size of the monodromy orbits on those spaces in geometric, rather than measure- 
theoretic terms. 

Our first result is of essentially topological nature. Before we state it we will need to 
introduce some notation. Let as before / : X — > B be a smooth holomorphic family of genus 
g curves and let o G B be a base point. Let X Q be the fiber of / over o and let 

map : iri(B,o) —> Map(X ) C Out(7Ti(X ) 

be the corresponding geometric monodromy representation. Here Map(X G ) denotes the map- 
ping class group of X Q . By definition Map(X c ) := 7r (Diff + (X o )) is the group of connected 
components of the group of orientation preserving diffeomorphisms of X . Alternatively 
Map(X D ) can be identified with the subgroup of index two in 0ut(7Ti(X o ) consisting of all 
outer automorphisms which act trivially on if 2 (X ,Z). Fix some topological double cover 
v : X Q — >• P 1 and let i G Map(X Q ) be the mapping class of the covering involution. The 
hyperelliptic mapping class group of X Q is defined to be the centralizer A(X ) of % in Map(X D ): 

A(X Q ) := {0 G Map(X o )|0z0- 1 = %}. 

Note that the definiton of A(X G ) depends on the choice of the double cover v and so the hy- 
perelliptic mapping class group is defined as a subgroup in Map(X c ) only up to conjugation. 
We will say that the geometric monodromy of / dominates the hyperelliptic monodromy if 
up to conjugation in Map(X ) the monodromy group mon(7Ti(S, o)) C Map(X Q ) contains a 
subgroup of finite index in A(X ). 

Theorem A Assume that the monodromy of f dominates the hyperelliptic monodromy, e.g. 
assume that the image mon(7Ti(-B, o)) C 0ut(7Ti(X o )) is of finite index in 0ut(7Ti(X o )). Then 
there exists a positive integer g so that if g > go and n is any fixed odd integer, we have: 

(i) There is no meromorphic function on Ms(X ,ra) an which is invariant under the action 
o/mon^(7r 1 (i?, o)) (equivalently there is no meromorphic function on M DR (X / B , n) an 
which is V}) R -invariant) ; 
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(ii) In the case of Mb(X ,ti), considered with its natural structure of an afftne algebraic 
variety, there exists a point 6 M^(X ,n) so that the orbit 

mon^(7r 1 ( J B,o)) ■ x B C M B (X Q ,n) 

is Zariski dense in M B (X ,n); or in the case of Mbr{X/ B,n) — > B there is a leaf of 
the foliation defined by V^xr which is Zariski dense in the algebraic Zariski topology. 

This theorem suggests that for families / : X — > B with a "large enough" geometric 
monodromy one should expect Zariski dense monodromy actions on non-abelian cohomology. 
Geometrically families with large monodromy naturally arise from hyperplane sections and 
Lefschetz fibrations. In this context we prove the following 

Theorem B Let Z be a smooth projective surface with b\{Z) = 0. Let Oz{^-) be an ample 
line bundle on Z and let n > 1 be a fixed odd integer. Then there exists a positive integer I 
(depending only on Z and Oz{^) ), such that for every k > I and for every Lefschetz fibration 
f : Z — > P 1 in the linear system \Oz{k)\ we have: 

(i) There is no meromorphic function on MB{Z ,nY n which is invariant under the action 

of mon^(7ri(P 1 \ {pi, . . . , p^}, o)) (equivalently there is no meromorphic function on 
MDRiZ/F 1 \ {p x , . . . , Pfi }, n) an which is ^-invariant); 

(ii) In the case of Mg(Z , n) ; there exist a point xb £ M B (Z ,n) so that the orbit 

mon B (n 1 (¥ 1 \{p 1 ,...,p fl },o)) ■ x B C M B (Z , n) 

is Zariski dense in M B (Z , n); or in the case of the space M DR (Z /F 1 \ {pi, . . . ,Pfj}, n) 
the foliation defined by has a Zariski dense leaf. 

Here as usual Z is the blow-up of Z at the base points of the pencil and Pi, ■ ■ -P^ £ IP 1 are 
the points where the map Z — » P 1 is not submersive. 

These statements can be viewed as nonabelian analogues of Deligne's irreducibility the- 
orem [Deligne80l Section 4.4] and |J anssen83] . which asserts that the monodromy group on 
the first cohomology of a Lefschetz pencil of curves is a subgroup of finite index in the full 
symplectic group of the lattice of vanishing cycles. 

The paper is organized as follows. In Section we examine the action of a finitely generated 
group on an affine algebraic variety. We show how the existence of a Zariski dense orbit can be 
deduced from the existence of an open orbit for the linearized action on the tangent space at 
a fixed point. Section describes a particular point in the moduli space of representations 
of the fundamental group of a curve X , which corresponds to the Schrodinger representation 
of a suitably chosen finite dihedral Heisenberg group. This point is smooth and fixed by a 
subgroup of finite index in the monodromy. Moreover the tangent space of the moduli of 
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representations at the 'Schrodinger point' is naturally identified with the first cohomology 
of an etale cover Y Q of X . Finally in Section ETT1 we discuss the necessity of the hypotheses 
of Theorem 1X1 and iBl for the existence of a dense monodromy orbit. We conjecture that the 
density holds under very mild assumptions and give some additional evidence supporting 
the conjecture. 

Acknowledgments. We would like to thank A. Beilinson, P. Deligne and I. Smith for 
some very enlightening conversations. We are very grateful to B. Toen, C. Walter and to 
the referee for pointing out some mistakes in a preliminary version of this paper and for 
suggesting improvements in the exposition. The second and third authors would like to 
thank UC Irvine for their hospitality during the month of July of 1998, when most of the 
ideas for the present paper took shape. The first and the second author wish to thank the RiP 
program of the Mathematical Forschunginstitut Oberwolfach and the Volkswagen-Stiftung 
for their support and the excellent working conditions during two weeks in the Summer of 
1999, when a substantial part of this work was done. 

2 Preliminary reductions 

We start with some general results about linear group actions on algebraic varieties, which 
will allow us to localize at a point the Zariski density property of an action. 

2.1 Open orbits and dense actions 

Suppose M is a reduced irreducible affine scheme of finite type over C. Write M = Spec(A) 
and let T be a finitely presented discrete group acting on M by algebraic automorphisms. 
Thus T acts on A by C-algebra automorphisms. 

Lemma 2.1 If there is one point in M whose orbit under T is Zariski dense, then there is a 
countable union of proper closed subvarieties of M such that for any x not on this countable 
union, the orbit of x is Zariski dense. 

Proof. The fact that A is of finite type over C means that we have a surjection from a 
polynomial ring to A. In particular (doing an enumeration of the monomial basis for this 
polynomial ring) we obtain a filtration of A: 

oo 

A = [jA l 

i=0 

by finite-dimensional sub-C- vector spaces A{ C A. 

For each pair of integers i, k > let G' ik denote the space of fc-tuples of elements of Ai. 
It is a finite dimensional affine space. For V <E G' ik let Ia{Y) denote the ideal in A generated 
by V — ( v ij ■ ■ ■ j v k) an d let Z v d M denote the reduced closed subvariety of M defined by 
V. 
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There is a closed algebraic subset 

Z' hk c G' hk x M 

such that for each V G G' ik the fiber over V is equal to Zy. To see this, note that the 
coordinate ring of G\ k is the symmetric algebra on the dual of Af k . The coordinate ring of 
G\ k x M is thus the tensor product and the projections-inclusions Af k — > A can be viewed 
as elements of this coordinate ring; they generate the ideal of the closed subset Z' ik . Let 
Gi t k C G' i k be the complement of the origin and let Z^ k be the inverse image of G^ k . The 
family 

Z iJk C G hk x M 

parameterizes precisely the closed proper subvarieties of M which are cut out by ideals 
generated by fc-tuples of elements in A { . (Note that since Spec (A) is reduced and irreducible, 
any non zero fc-tuple generates the ideal of a proper subvariety). 
For each 7 G T we obtain the translate 

lZi,k C G ijk x M. 

Let Gi,fc(7) C G i>k denote the subset of points V such that ^Z v = Z v . We claim that this 
is a constructible subset. It may be described as the set of points V G G i>k such that the 
intersection fl Z^ k fl ({V} x M) contains Z i>k fl ({V"} x M). This set is the complement 
in Gi t k of the image of the map 

(Zi jk — ijZ i)k fl Z iik )) — > G ijk 

so it is constructible. 

Now as 7j runs through a finite set of generators we obtain the intersection of this finite 
collection of subsets, which is again a constructible subset 

G ilfc (r) :=f|G i>Jfc ( 7i )cG <>fc . 
3 

Each V in G,^ k (T) defines a T-invariant closed proper subvariety Zy C M, and conversely 
it is clear that any T-invariant closed proper subvariety of M appears as a Z v for some i, k 
and some V G Gi tk (T). 

Let iVj^ be the union of the points contained in all of the the subvarieties Zy for all 
V G G ijk (r). This is again a constructible set since it is the image of the projection 

Zi, k x Gt k Gi, h (r) -+ m. 

If a point x G M is contained in any proper closed T-invariant subvariety then it is contained 
in some N itk . Note also that N itk is T-invariant; indeed it is a union of the T-invariant subsets 
Zy for all the V G G i>fc (r). 

We now have two possibilities: either 
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(a) one of the constructible subsets N ijk is dense in M; or 

(b) all of the constructible subsets are contained in proper closed subvarieties N^- 

In case (b) , we obtain a countable union of closed subvarieties [j { k N itk such that if x G M 
is a point which is not in this countable union, then x is never contained in a proper closed 
T-invariant subvariety. 

In case (a) we claim that no point has a Zariski dense orbit. Indeed the complement of 
the constructible set which is dense, has a closure which is itself a proper Zariski closed and 
T-invariant subvariety (note that all of our constructible sets were T-invariant). Thus any 
point here has non Zariski dense orbit. On the other hand, any point in the complement of 
this closed set is in the open interior of the constructible set in question, so it is by definition 
in the image of one of the Zy, i.e. it is in a proper T-invariant closed subvariety. This proves 
that in case (a) no point has a Zariski dense orbit. 

Assume now that there is some point in M which has a Zariski dense orbit, then we are 
in case (b), so there is a countable union of closed subvarieties such that if x is not in here 
then x has Zariski dense orbit. The lemma is proven. □ 

We are now ready to prove the main result of this section 

Theorem 2.2 Suppose that V acts on an irreducible complex affine algebraic variety M. 
There are two possibilities: either 

(1) there exists a nonconstant T -invariant meromorphic function; or 

(2) there exists a point x G M with Tx Zariski- dense in M. 

Proof. Assume that no point in M has a Zariski dense orbit. This means that we are in 
the situation (a) discussed in the proof of Lemma 12.11 In other words, there exist integers 
i,k > so that the T- invariant constructible set Ni t f. C M is dense in M. To simplify 
notation put S := G^fT) and Z := Z i k x G . k G i>fc (r). By construction S and Z are both 
schemes of finite type over C and the natural maps 

Z *■ M 

S 

constitute a T equivariant family of closed T-invariant proper subvarieties of M. Moreover 
the total space Z of this family maps onto the T-invariant constructible subset N^k C M. 

Let x be a point in the open interior of iV^ C M and let Z C M be the Zariski closure 
of Tx. Passing to a subgroup of finite index in T we can assume that Z is geometrically 
irreducible. Let V G S be such that Z = Zy. Then x G Zy and by further localizing S we 
may assume that all the fibers of Z — > S are irreducible and of the same dimension. 

Let W C Z denote the set of all points which are contained in two or more distinct fibers 
Zy x and Zy 2 (i.e. two fibers with Zy l ^ Zy 2 ). We claim that this is a constructible subset. 
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To see this, look at the closed subvariety 2x^2c2x2 and the subset I of S x S given 
by the condition 

/ := {(Vi, V2) G S 1 x 5 I such that Zy 1 = Zy 2 as subvarieties in M } . 

Note that I C 5 x 5 is a constructible subset (see this by taking a compactification of M, 
compactifying the family Z — > S 1 and then using Chow schemes) and so the preimage X of 
J in Z x Z is also constructible. The subset W is the projection of Z Xm Z — I on one of 
the factors so is constructible. 

Next we claim that W does not contain our original point x thought of as a point in the 
fiber V G S. For if it did, this would mean that there was a distinct Zy 7^ Zy containing x, 
and then x would be contained in the T- invariant set Zy* C\Zy\ but this latter set is a proper 
subset of Zy, contradicting the fact (by definition of our family) that Tx is Zariski-dense in 

Zy. 

It follows that there is an open set of the total space Z which does not meet W. Note 
that it is clear from the definition that W is the inverse image of a constructible subset in 
M; thus this subset does not contain the generic point of M so there is a closed set C such 
that W is contained in the preimage of C in Z. Since W is a T-invariant set, we may replace 
C here by the intersection of all of its translates so we can assume that C is T-invariant 
as well. Finally then we can throw C out of M (i.e. replace M by M — C in the whole 
discussion) so we may assume that W is empty. Note that the new M will no longer be 
affine but only quasi-affme. This does not affect the rest of the argument though since the 
existence of meromorphic functions can be detected on opens. 

By taking a compactification of M and looking at the Chow scheme of subvarieties of 
this compactification, we can replace our family by a family indexed by a new base scheme 
S where each fiber (considered as a subset of M) occurs exactly once. 

Now by the above reduction the morphism Z —>■ M is injective on points; also its image 
hits a generic point of M. Thus there is a largest open subset of M over which this is an 
isomorphism and we can replace M by this open subset (which is T-invariant). Hence we 
obtain a T-invariant fibration M — >• S. Now any meromorphic function on S pulls back to 
give a T-invariant meromorphic function on M. This essentially proves the theorem. The 
only problem we need to address is that in the construction of the family Z — >• S we had to 
pass to a finite index subgroup of T and so the function just constructed may be invariant 
only under a subgroup of finite index of T. This however is easily remedied - by taking 
the different invariant polynomials in the Galois translates of our meromorphic function we 
obtain a meromorphic function invariant by the full T. □ 

In view of Theorem 12.21 we need to find effective criteria for the non-existence of invariant 
rational functions on an affine variety. One such criterion is given by the following lemma: 

Lemma 2.3 Let T be a finitely presented discrete group. 

(alg) Suppose M is a reduced irreducible scheme of finite type over C. Suppose that F acts on 
M by algebraic automorphisms and lety e M be a point in the smooth locus of M , fixed 
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by the action, so that T acts linearly on the tangent space T y M. Let G C GL(T y M) be 
the Zariski closure o/im[r — > GL(T y M)\. Assume that G acts on T y M with an open 
orbit, and that the connected component G° of G has no nontrivial characters. Then 
there is no V -invariant rational function on M. 

(an) Suppose that N is an irreducible analytic space on which Y acts by analytic automor- 
phisms. Suppose that y & N is a point in the smooth locus of N, fixed by the action, 
so that T acts linearly on the tangent space T y N. Let G C GL(T y N) be the Zariski 
closure of im[T — > GL(T y N)]. Assume that G acts on T y N with an open orbit, and 
that the connected component G° of G has no nontrivial characters. Then there is no 
Y -invariant analytic-meromorphic function on N 

Proof. Clearly the statement of the lemma is insensitive to passing to a finite index subgroup 
of r and so we may assume that G = G°. We will only give the proof in the algebraic case. 
The analytic case is completely analogous. 

Suppose that h is such a function, and write the germ of h at y as f/g with /, g G Om, v 
relatively prime. Then for any 7 G T, 

L = h = 1 *h= — , 
9 1*9 

which implies that there is a unit u(j) G 0\ ly with 

i*f = u(i)f, 

1*9 = u(j)g. 

Note that 7 1— > u(j) is a cocycle for T acting on the multiplicative group of units 0^ y . In 
particular we get that the value u(j)(y) is a character of T. 

Without loss of generality we may assume that h is not an invertible function at o 
(otherwise subtract the constant function with the same value at o) hence we may assume 
that one of / or g has a nontrivial leading term of some degree. We may suppose that / has 
such (otherwise replace h by h' 1 ). Let f m be the leading term of /. Note that f m (of degree 
m) — modulo higher order terms — is an element of Sym m T v (M) y . The action of T on this 
leading term factors through the group G. Our previous formula gives 

J*(fm) = u(j)(y)f m . 

This shows that 7 1— > 14(7) (y) comes from a character of G; but by assumption there are no 
such characters Therefore we get 7*(/ m ) — fm, so f m is a G-invariant homogeneous form of 
degree m. 

In particular, we can think of f m as a G-invariant polynomial function on the tangent 
space T y M. This contradicts the supposed existence of an open orbit in the action of G on 
T y M. The lemma is proven. □ 

The essential consequence of Theorem 12.21 that we need is the following localization state- 
ment. 
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Corollary 2.4 Suppose that y G M is a point in the smooth locus of M, fixed by the action, 
so that T acts linearly on the tangent space T y M. Let G C GL(T y M) be the Zariski closure of 
im[r — > GL(T y M)]. Assume that G acts on T y M with an open orbit, and that the connected 
component G° of G has no nontrivial characters. Then there exists a point x G M with Fx 
Zariski- dense in M. 

Proof: By Theorem l2.2l we only have to rule out the possibility that M admits a nonconstant 
T-invariant meromorphic function. This however is precisely the content of Lemma l23K a lg)- 
The Corollary is proven. □ 

Theorem 12.21 Lemma 12.31 and Corollary 12.41 not only provide a convenient localization 
criterion for the density of an action but also suggest another geometric notion of 'largeness' 
of the T-orbits. Motivated by Theorem 12.21 we define various degrees of analytic generic 
Zariski denseness (AGZD for short) as follows: 

Definition 2.5 Suppose that a finitely generated group T acts by analytic automorphisms 
on an irreducible analytic space N. Let m : V x iV — > iV be the action map. 

• We say that the action m is AGZD1 if there is no Y -invariant analytic meromorphic 
function f on N. 

• We say that the action m is AGZD2 if there is no pair (U, f), where U C N is a T- 
invariant analytically Zariski dense open subset of N and f : U — > Z is aT equivariant 
holomorphic map from U to a complex analytic space Z with dimZ < dimiV. 

• We say that the action m is AGZD3 if there is a point x G iV such that m(T x {x} 
is analytically Zariski-dense in N. 

• We say that the action m is AGZD4 if there is an analytically Zariski dense open 
subset U C N such that for every x G U the V orbit of x is analytically Zariski-dense 
in N . 



Clearly for an analytic action m one has the implications: 

AGZD4 =4> AGZD3 => AGZD2 => AGZD1, 

but we don't think that the converse implications are true. Similarly, it is clear that if m is 
actually an algebraic action, then AGZD1 implies that m is Zariski-dense in the algebraic 
sense of Theorem 12.21 

Suppose now that B is a base scheme and that p : M — > B is a morphism equipped 
with a connection V (by which we mean a stratification over the crystalline site of S 
Crothendieck68j . |Simpson 95|). For the following definition it is not necessary to assume 
that V is integrable. 
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Definition 2.6 Suppose that B and the generic geometric fiber ofM/B are irreducible. We 
say that (p : M — > B, V) generically Zariski dense (or GZDj &/ t/iere is no algebraic 
meromorphic function f on the total space M which is invariant under V. 

If the connection V is integrable, then the corresponding analytic family is associated 
to a local system of complex analytic spaces over B, which in turn corresponds to the 
monodromy action m of T := tti(B, o) on a fiber M* n . It is clear that if m is AGZD1, then 
(p : M -> 5, V) is GZD. 

In particular, the AGZD1 property for Mb(X ,u) or equivalently Mdr(X q , n) implies 
the algebraic generic Zariski-denseness property GZD for the Gauss-Manin connection V^. 

Consider now a family of smooth projective connected curves / : X —>■ B and let o G B 
be a base point. We will show that when the geometric monodromy of / : X —>■ B is of finite 
index in the mapping class group or when / comes from a Lefschetz pencil as in Theorem [Bj 
then the monodromy action of tti(B,o) on M#(X ,n) is AGZD1. In combination with 
Theorem 12.21 this fact yields statement (i) of Theorems [X] and El As explained above this 
automatically gives the analytic statement (ii) in both theorems. In fact, it follows from the 
above considerations that M£>R(X/B,n) — > B together with the non-abelian Gauss-Manin 
connection is GZD in the sense of Definition 12.61 

In view of all this it only remains to show that under the hypothesis of Theorems El or 
|B]the algebraic action 

mon^ : 7n(S,o) -> Aut(M B (X ,n)). 

on the affine variety Mb(X q , n) is AGZD1. (Note that since X Q is a smooth curve the variety 
Mb(X , n) is irreducible by Simpson95, Section 11].) In view of Corollary 12. 41 to achieve this 
we only need to find a smooth point p G Mb{X q , n) which is fixed by the monodromy group 
mon^(7Ti(S, o)), and for which the Zariski closure of mon^(7Ti(S, o)) C GL(T[p]Mb(X , n)) 
acts on T[ rho }MB(X , n) with an open orbit and has a connected component of the identity 
which admits no non-trivial characters. 

In the next section we describe a proposal for such a point p which utilizes the Schrodinger 
representation of a finite dihedral Heisenberg group. Later on, we will show in Sections 13.11 
and 13.21 that the open orbit property for the monodromy action on the tangent space at p 
holds, provided that the geometric monodromy of / : X — > B is large enough. 

We have stated the additional properties AGZD2-4 in order to pose the question: which 
of these properties hold for families whose monodromy has finite index in the mapping class 
group? For (sufficiently ample) Lefschetz pencils? 

2.2 The Schrodinger representation 

Since p is supposed to be fixed by the monodromy a natural choice would be to take p to 
be the trivial representation of 7Ti(X ) in GL(n, C). However the trivial representation is a 
singular point of M B (X ,n) and so is unsuitable for our purposes. On the other hand any 
representation 

p:7n(JQ->GL(n,C) 
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which has finite image will be fixed under some finite index subgroup of mon^(7r 1 (i?, o)). 
Furthermore, the properties AGZD1-AGZD4 and GZD are obviously stable under passage 
to a finite index subgroup of T. Hence we are free to replace B by any finite etale cover of B, 
and so it is enough to find a finite representation p which satisfies the open orbit condition. 

In order to apply Corollary 12.41 we also need to choose p G Mb(J ,ti) to be a smooth 
point. This is equivalent to choosing p to be an irreducible representation. 

To construct such a representation we proceed as follows. Let \x n C C x be the group of 
all n-th roots of unity. Let Z/n := Hom^(Z/n, C x ) denote the group of characters of the 
cyclic group Z/n. Consider the finite Heisenberg group H n . By definition H n is the central 

extension 

-> n n -> H n -> Z/n x Z/n -> 

corresponding to the cocycle e : (Z/n x Z/n) 2 — > /x„ C C x , e((a, a), (a', a')) = a'(a). 
Explicitly can be identified with the set [i n x Z/n x Z/n with a group law given by 

(2.1) (A; a, a) ■ (A'; a', a') = (AA'a'(a); a + a', «a'). 



Let (fi n : if n — > GL(V n ) be the Schrodinger representation of H n |Mumford93j . By definition 



(fi n is the unique n-dimensional irreducible representation of H n which has a tautological 
central character. One way to construct (fi n is to observe that the natural injective map 

/Lt n x Z/n ^ H n , (A, a) h-> (A; 0, a) 

is a group monomorphism, i.e H n contains fi n x Z/n as an abelian subgroup. Let T be 

the one dimensional complex representation of fi n x Z/n which corresponds to the pullback 

of the tautological character of under the projection fi n x -^A 2 ~~ > A*n- m other words 
T = (C, r) where r : /x rt x Z/n — >• C x is given by r(A, a) = A. In terms of T then we have 

Explicitly we can identify V„ with the vector space of all complex valued functions on the 
finite set Z/n and the action (fi n by the formula 

[4> n (\;a,a)f](x) = Xa(x)f(x + a), 

for all x G Z/n, / G V n and (A; a, a) G 

The irreducibility of the representation (fi n follows from Frobenius reciprocity or directly 
by noticing that V n has a basis consisting of the characteristic functions of the elements in 
Z/n and that the subgroup Z/n C H n acts transitively on the elements of this basis. In 
particular if we compose (fi n with some surjective homomorphism TTi(X ) -» H n we will get a 
representation of ttx(X ) in GL(V n ) which is irreducible and has finite image. Unfortunately, 
it turns out (see Remark 12.1 (J I) that this representation can not be used directly to obtain an 
open orbit action on the tangent space to Mb{X ,ti). However a slight modification of this 
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representaion does the job. The modification involves an extension of H n of dihedral type 
which we proceed to describe. 

Let n 2 = {~ 1) +1} C C x be the group of square roots of one. The group fx 2 acts naturally 
on Z/n x Z/n as the inversion on both factors. This action clearly preserves the cocycle 
defining the Heisenberg central extension — > \i n — > H n -* Z/n x Z/n -> and so we get a 

natural action of /x 2 on H n . Explicitly, if we think of H n as the set |i„xZ/nxZ/ti equipped 
with the group law (|2.1)1 . then an element e G /x 2 acts on H n via (A; a, a) i— > (A, ea, a e ). We 
define the dihedral Heisenberg group as the semidirect product 

DH n := /x 2 K # n 

for the above action. Thus DH n can be identified with the set \i 2 x /Lt n x Z/n x Z/n with a 
group law given by 

(2.2) (e, A, a, a) • (e', A', a', a') = (ee', XX'a' £ (a), a + ea', aa' £ ). 

In particular, for each (e, A, a, a) G DH n we have 

(e, A, a, a) = (1, A, a, a) ■ (e, 1, 0, 1 ) 

= (e, 1,0, 1) • (1, A,ea,a e ), 

where 1 : Z/n — >• C x stands for the trivial character, i.e. 1(a) = 1 for all a. 

Observe next that the Schrodinger representation <fr n : H n —>■ GL(V n ) extends naturally 
to a dihedral Schrodinger representation 

O0 n : M n -> 

defined by 

(0^ n (e, A, a, a)f)(x) = Xa(x)f(e(x + a)) 

for all / G and all x G Z/n. 

Recall that if C is any smooth curve of genus g, then there is a surjective homomorphism 
7Ti(C) -» i^g onto a free group of g generators. This homomorphism is obtained by moding 
7Ti(C) out by the normal subgroup generated by the a-cycles for a standard basis in the first 
homology of C. Note furthermore that DH n is generated by the three elements (—1, 1,0, 1 ), 

(1, 1, 1, 1) and (1, 1, 0, a), where a G Z/n is any generator. Hence we can find a surjective 
homomorphism tti{C) — > F g — > DH n as long as g > 3. 

By hypothesis the genus of X Q is big enough and so we can find a surjective homo- 
morphism ip n : tti(X ) -» S)if„. Let p : 7Ti(X ) — > GL(V^) denote the composition 
p := ip n o O0 n . By construction p is irreducible and so represents a smooth point of the 
moduli of representations. Moreover by a standard deformation theory argument (see e.g. 
|Lubotzky-M agid85|) we can identify the Zariski tangent space T[ p ]Mb(X , n) with the group 
cohomology H 1 (7Ti(X ),ad(p)), where 

ad(p) : TTxpQ -> GL(End(K)) 
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is the natural representation induced from p. Explicitly ad(p) = <S> 50 n ) ° VVi an d since 
ad(0„) = (fin® (fin has a trivial central character we see that ad(p) factors trough the quotient 

group DH n -» fj, 2 K (Z/n x Z/n). 

It is not hard to calculate i/ 1 (7Ti(X ), ad(p)) in terms of geometric data on the curve X Q . 
The action of /x 2 on Z/nx Z/n induces an obvious action (inversion on both factors) of /x 2 
on the group characters 

Hom(Z/n x Z/n, C x ) = Z/n x Z/n. 

Now for each orbit u £ (Z/n x Z/n)//i 2 we an irreducible representation W u of 
/x 2 x (Z/n x Z/n). The representation VF( J)0 ) corresponding to the trivial character is just 
the trivial one dimensional representation of fi 2 x {Z/n x Z/n). For any other orbit u we 
have that u = {x^X 1 } f° r some non-trivial character x £ Z/n x Z/n and so W u is the 
representation of fi 2 x (Z/n x Z/n) induced from the one dimensional representation (C, x) 
of Z/n x Z/n. Thus = (C, x) © (C,x _1 ) as a representation of Z/n x Z/n and the 
generator of \x 2 acts by switching the two summands. In particular W u is a two dimensional 
irreducible (we are assuming that n is odd here) representation of fi 2 x (Z/n x Z/n). 
With this notation we have 

Lemma 2.7 The tangent space to MB(X a ,n) at the dihedral Schrddinger representation p 
is given by 

(2.3) T [p] M B (X ,n) = H\X ,ad( P )) = H\X , W u ) 

u£(Z/nxZ/n)//j 2 

where W u is the local system on X Q corresponding to the representation 
TTipQ ^ ©fl- n — > fj, 2 k (Z/n x Z/n) -> GL(W / " U ). 

Proof. Note that a representation k : Z/n x Z/n — > GL(V) of the abelian group Z/n x Z/n 

will extend to representation of the 'dihedral' group n 2 x (Z/n x Z/n) if and only if k is 
self-dual. Furthermore, each self-dual representation k has a canonical dihedral extension: 

: /Lt 2 x (Z/n x Z/n) -> GL(V), 

in which /x 2 acts as the self-daulity automorphism of V. Concretely if we decompose (V, k) 
into a direct sum of characters of Z/n x Z/n, then the self-duality of V will identify the 
multiplicity space of each character x with the multiplicity space of the character x 1 - 111 
particular the multiplicity spaces in the character decomposition of (V, k) depend not on 

the individual characters but rather on the /Lt 2 -orbits u G (Z/n x Z/n)//x 2 . Hence (V, 0/c) 
decomposes as 

(V,i>K)= W^®M n , 

«e(Z/nxZ/n)//i 2 
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where M u denotes the multiplicity space of a character x G w in (V, k). 

Consider now the representation ad(0 n ) of H n . Since it has a trivial central character, it 
factors through a representation 

ad(0 n ) ab : Z/n x Z/n ^ GL(End(K))- 

This is the abelian part of ad(0 n ). The representation ad(0 n ) ab is self-dual by construction 
and so admits a canonical dihedral extension 

Oad(0 n ) ab : fi 2 x (Z/n x Z/n) -> GL(End(K))- 

This dihedral extension fits in the commutative diagram 

7n(X )-^GL(End(V B )) 

Oad(<?> n ) ab 

M n /x 2 x (Z/n x Z/n) 

and so understanding ad(p) is equivalent to understanding ad(0„) ab . But ad(0 n ) ab is just 
the regular representation of the abelian group Z/n x Z/n. To see this note first that 
dimEnd(K l ) = n 2 = dimC[Z/n x Z/n]. Now since every irreducible representation of 
Z/n x Z/n occurs in C[Z/n x Z/n] with multiplicity one we need only to check that for every 
character x '■ %/n x —> C x we have 

dimHom (z/nx ^ n) _ mod ( X ,ad(0 n ) ab ) > 1. 

But the group of characters of Z/n x Z/n is naturally isomorphic to Z/n x Z/n and so each 

character x as above is given by a pair (£, x) G Z/n x Z/n via the formula x( a , a ) — £,( a ) ma { x )- 
Therefore we only need to show that for any pair (£, x) there exists a non zero element 
G End(K) = K V ® V n so that 

(a,a)i4 KiX) = ^(a)a(x)A { ^ x) 

for all (a, a) G Z/n x Z/n. 

To construct the element A^ x ^ recall that the vector space V n = C[Z/n] has a natural 
basis {e , ei, . . . , e n _{\ consisting of characteristic functions of elements of Z/n, i.e. et(j) : = 
5ij. Let {e^ , e(,..., e^} be the dual basis of V// . Then in terms of the basis {e/ ® e 3 -}™ji 
of V// ® \4 the representation ad(0 n ) ab is given by the formula 

[ad(0 n ) ab (a, a)](e/ <8> e,) = a(j - i)et a <8> e,_ a . 



In view of this we may take 



n-1 



i=0 
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which is obviously a non-zero eigenvector corresponding to the character \ = (£> x )- 
This shows that 

(End(K),ad(0 n ) ab ) = C[Z/n xZ?n] = ® xe£JnxZ/n (C, X ), 

and so the lemma is proven. □ 

Let us now go back to the problem of checking whether mon(7Ti(5, o)) has a dense 
orbit on Mb(X , n). As mentioned above, the fact that p has a finite image implies that 
the conjugacy class of p will be fixed by some finite-index subgroup of mon(iri(B, o)). In 
particular, applying Corollary 12.41 to this subgroup, it follows that in order to show the 
existence of a dense mon(7Ti(5, o))-orbit on M B (X a , n), it suffices to check the following two 
items: 

(i) The Zariski closure G of 

im[mon(7r 1 ( J B,o)) -> GL(H 1 (X , ad(p)))] 
in GL{H 1 {X 0) ad (/?))) has an open orbit on i? 1 (X , ad(p)). 

(ii) The identity component G° of G does not have any non-trivial characters. 

Condition (ii) follows easily from the isomorphism (J2.3)) : 
Lemma 2.8 The identity component G° of the Zariski closure of 

im[mon(7n( J B,o)) -> GL(H 1 (X , ad(p)))] 
in GL(H 1 (X Q , ad(p))) /ias no non-trivial characters. 

Proof. Indeed, let p : Y Q — >• X Q be the dihedral Galois cover of X Q corresponding to the 
surjection 

(2.4) tti(X ) DH n -> ^ 2 x (Z/n x Z/n). 

Then Y" is a smooth connected curve and the pushforward of the trivial one dimensional 
local system Cy D via p is precisely 

P*C Yo = W u . 

u6(Z/nxZ/n)//i 2 

Thus we can identify -ff 1 (X , ad(p)) with if 1 (F D ,C). 

Next observe that without a loss of generality we may assume that the family / : X — > B 
has an algebraic section a : B — > X. Indeed, since X is quasi-projective a generic hyperplane 
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section on X will be a multisection of /. But replacing B by an etale cover of a Zariski open 
set of B will replace mon(7Ti(5, o)) by a subgroup of finite index. In particular such a 
replacement will not affect the property that mon(7Ti(.B, o)) fixes the conjugacy class of p. 
In fact, by taking another etale cover if necessary we can ensure that not only the conjugacy 
class of p is fixed under mon(7Ti(S, o)) but that the actual representation 

p:Trx(X ,a(o))^GL(V n ) 

remains fixed under mon(7r 1 (5, o)). Indeed, to achieve this we only need to pass to the finite 
index subgroup of the monodromy which preserves the kernel of p and acts trivially on the 
finite group im(p) = DH n . 

Assume that we are in this situation. Then p lifts to a well defined representation of 
7Ti(B,o) K mon TTi(X ,a(o)) and so defines a DH n -cover of X. In particular under the iden- 
tification H\X , ad(p)) = H X (Y , C) the action of mon(7Ti(.B, o)) on H X {X , ad(p)) becomes 
just the monodromy action of tti(B,o) on H l (Y a ,C) corresponding to the family of curves 
Y — > B. But by Deligne's semisimplicity theorem |Deligne72"l Corollaire 4.2.9] the mon- 
odromy action on the middle dimensional cohomology of any smooth projective family over 
a quasi-projective base has a semisimple Zariski closure. Thus G° must be a connected 
semisimple algebraic group and so has no non-trivial characters. The lemma is proven. □ 



In order to check that condition (i) is satisfied we need to make sure that the monodromy 
group of the family / : X — * B is sufficently large. 

Before we explain how this is achieved we need to introduce some notation. On the way 
we will also rephrase the condition (i) in a slightly more general context. 

Let / : X — > B be a smooth family of connected curves of genus g > 3. Let as before 
o G B be a fixed base point and let 

mon : m(B,o) -> Map(X ) C Out(7Ti(X )) 

be the corresponding geometric monodromy representation. 

If / : X — > B has a holomorphic section a : B — ► X the representation 

mon : 7Ti(B, o) — > M&p(X ) 

can be lifted to a geometric monodromy representation respecting the base point: 

mon CT : 7ti(B, o) -> Map 1 ^) C Aut(7r 1 (X D , er(o))). 

Here Map 1 (X ) denotes the mapping class group of the once punctured surface X D — {a(o)}. 

Fix a finite abelian group A and let DA := /x 2 ix A denote the standard dihedral ex- 
tension of A in which the generator (—1) G /x 2 acts as the inversion on A. Fix a surjective 
homomorphism 7r 1 (X D , c(o)) -» DA and let p : Y® A — ► X be the corresponding Galois 
cover. Let Mav 1 {.X ,DA) be the group of p Sj4 -liftable mapping classes, i.e. 



Map\X ,DA) = |<p G Map x (X ) 



cp preserves ker[7r!(X , a(o)) -» DA] and <p 
induces the identity on DA 
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Clearly Map 1 (X a , D A) C Map 1 (X G ) is of finite index and consists precisely of the map- 
ping classes on X a which lift to mapping classes on Y® A . Furthermore, since by definition 
each ip G Map 1 (X a , DA) induces the identity on DA it follows that any lift <p G M&p(Y® A ) 
of ip commutes with the action of DA on Y® A , and that DA acts transitively on the set of all 
such lifts. Thus, if we define LMap 1 ^, DA) C Map 1 (Y^ A ) to be the sub group consisting 
of all lifts of elements in Map 1 (X ) we see that LMap\X , DA) fits in a short exact sequence 
of groups 

1 — > DA — > LMap^Xo, DA) -> Map 1 ^, ©A) -> 1. 

Assume now that mon <T (7Ti(.B, o)) C Map 1 (X , 2) A). In particular mon°~ (ni(B , o)) preserves 
it\{Y® A ) and so we get a short exact sequence of groups 

1 -> ^(5,0) X monCT TTi^) -> ^(^,(7(0)) — > DA — > 1. 

Let y Sj4 — > X denote the DA-Galois cover of X corresponding to the homomorphism 
7Ti(X, cr(o)) -» DA. By construction n 1 (Y' SA ) = tt 1 (B,o) x mou * 7r 1 (Y^ A ) and the corre- 
sponding monodromy representation 

mon si : ^{B,o) - Map(Y? A ) C 0ut(7r 1 (F o ;DA )) 

lands in LMap 1 (X , DA). 

Furthermore, note that from the viewpoint of the density properties we are interseted 
in, the conditions that / : X — > B has a section and that mon(7Ti (B,o)) C Map\X ,DA) 
are harmless. Indeed, as explained in the proof of Lemma 12.81 if / : X — > B is an arbitrary 
smooth projective family of curves with B smooth and connected and X quasi-projective, 
then we can always find a Zariski open set U C B containing the point o G B, and a 
finite etale cover (B',o') — ► (U,o), so that the pulled-back family X x B B' — > B' has a 
holomorphic section, and a geometric monodromy which is contained in Map 1 (X D , DA). 
Since 7Tx(U, o) -» it\{B, o) is surjective and 7Ti(B', o') C iti(U, o) is a subgroup of finite index, 
it follows that the geometric monodromy mon(7r 1 (i?', o')) of the family X x B B' — > 5' is a 
subgroup of finite index in mon(7r 1 (5, o)). In particular, any density statement we can make 
for the action of tti(B,o) will be equivalent to the corresponding density statement for the 
action of n\{B' , o'). 

The previous reasoning also shows that for any smooth family of curves 
f : X B, such that B is smooth and X is quasi-projective, and any surjective homo- 
morphism 7Ti(X ,cr(o)) -» DA, there is an appropriate (B',o') — > (B,o) and an DA-Galois 
cover F SA ^ X x B B', so that: 

• The image of the monodromy representation mon^A : fti{B',o') — > Map(Y® A ) is con- 
tained in LMap 1 (X ,DA); 

• The natural map mon^A(7Ti(B' , o')) —>■ mon(iTi(B,o)) has finite kernel and cokernel. 
Motivated by the discussion in Sections l2. II and l2~2l we make the following definition: 
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Definition 2.9 Let A be a finite abelian group. A pair (/ : X — > B,tti(X ) -» DA) is called 
good z/ t/ie Zariski closure of 

im [tti (£',(/) LMap 1 (X , DA) C Map(lf A ) -> Sp^Yf^, Z)) 

m Sp(ifi(y r :DA , C)) acts on Hi(Y® ,C) with an open orbit. 

Clearly now, the condition (i) is equivalent to the statement that if A = Z/n x Z/n and 
if the homomorphism 7i"i(X ) — > DA is induced from a surjective homomorphism 

7Ti(X ) -> 5>if„, 

then the pair (/ : X -> 5, tti(X ) -» DA) is good. 

In other words we need to find geometric restrictions on a family / : X — > B and a 
homomorphism 7Ti(X ) -» DA, which will guarantee that the pair 
(/ : X ► B, 7Ti (X ) -» DA) is good. 

As a first approximation one has to understand the image of LMap 1 (X , DA) into the 
symplectic group S~p(Hi(Y® A , Z). In the next section we will analyze the hyperelliptic part 
of this image for a suitably chosen surjection 7Ti(X ) DA. 



Remark 2.10 Satisfying condition (i) is a somewhat subtle task. In a preliminary version 
of this paper we attempted to work with a representation p : tci(X ) — ► GL(V^) which comes 
from a choice of a surjective homomorphism TTi(X ) -» H n onto the Heisenberg group H n 
rather than its dihedral extension DH n . This representation p is also irreducible and so 
gives a smooth point in Mg(X ,n) which satisfies condition (ii). Furthermore the image of 
LMap(X Q , A) into the corresponding symplectic group Sp(ifi(F G A , Z) was described explicitly 
by Looijenga |Looij enga97, Theorem 2.5]. Unfortunately the self-duality pairing on the 
representation ad(0 n ) ab gives rise to a quadratic function on H X {X , ad(p)) which will be 
preserved by all elements of the geometric monodromy and so we can not hope that G will 
have an open orbit for this choice of p. Replacing p by a representation coming from a 
surjection onto the dihedral Heisenberg group repairs this problem as we will see below. 
However this changes the setup and forces us to work with the two-dimensional dihedral 
representations W u instead of the characters of A. In particular this setup lies beyond the 
scope of Looijenga's analysis in |Looijenga97| and forces us to look for a description of the 
image of LMap 1 (X , DA) into the symplectic group Sp(-ffi(F I)A , Z)) based on first principles 
only. 

Remarkably enough, it turns out that such a concrete description is possible and that it 
leads to a stronger result which uses only hyperelliptic mapping classes to obtain an open 
orbit. However note that that we need to assume that n is odd in the explicit argument. 

3 Proofs of the main theorems 

In this section we prove Theorems 1X1 and FBI 
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3.1 The case of a hyper elliptic monodromy 



In this section / : X — > B will denote a smooth family of all hyperelliptic curves of genus g. 
For us the hyperelliptic curves will be represented as branched double covers of P 1 having 
2(7 + 2 branch points. From that point of view it is natural to take B to be the configuration 
space Conf 29+2 of 2g + 2 distinct points in P 1 . However, it is well known |Mess92| that no 
universal hyperelliptic family exists on that space. This can be remedied by either passing 
to an unramified double cover of Conf 2s+2 (see [Mess92, Remark 4]) or, alternatively, by 
taking an open subfamily of Conf 2£ , +2 . We take the second approach since it is better suited 
for our purposes. 

Concretely, we take B to be the configuration space of 2c? + 2 distinct points in the affine 
line C = P 1 — {oo}. To see that the universal hyperelliptic family on B exists, we only need 
to show that the incidence divisor 

E := { (fa, . . . , b 2g+2 }, x)EBx¥ 1 \xe {&!, . . . , b 2g+2 }} cBxF 1 

corresponds to a section o in the line bundle pj^ (9(2g + 2) onBx P 1 . Indeed, if this is the case 
we have a divisor a(B) in the total space tot(j>p 1 0(2g + 2)) of the line bundle p^ x O{2g + 2) 
and so X can be constructed simply as the preimage of cr(B) in tot(p^ ll O(g + 1)) under the 
natural squaring map. 

To see that S is in the linear system \p^ ll O(2g+2)\ one can argue as follows. Fix an affine 
coordinate z on C = P 1 — {oo}. For any integer k > let Sk '■= H°(F 1 ,Opi(k)). On the 
product Sk x P 1 we have the line bundle pp 1 0(k). Moreover the direct image Ps k *P^iO(k) 
is a vector bundle of rank k + 1 on Sk which is canonically isomorphic to Os h <8> Sk and so 
has a tautological section corresponding to ids k . Let Ok denote the corresponding section of 
PpiO(fc). By construction, the divisor of Ok consists of all pairs (s, x) 6 S^x P 1 with s(x) = 0. 
Consider now the subspace Ek C Sk of all monic polynomials in z of degree k. Since by 
definition B can be identified with the open subset of the afiine subspace E 2g+2 C 5 , 2 9 +2 
consisting of monic polynomials with simple zeros, we can take a = cr 2g + 2 \B- 

Therefore we have constructed a universal double cover 




over the configuration space B. Note that the fundamental group of B is the braid group 
B 2g + 2 on 2g + 2 strands and that the monodromy homomorphism for / : X — > B can be 
interpreted as the standard surjection from B 2g+2 onto the hyperelliptic mapping class group. 

Fix as base point o G B the double cover v Q : X Q — > P 1 with branch points 1,2,..., 2g+2 6 
1 C P 1 on the real axis. Make branch cuts Cj on P 1 from 2i — 1 to 2i along the real axis. 
Topologically, the surface X is obtained (see Figure ETTl below) by gluing together two copies 
of the sliced-up P 1 along the rims of the branch cuts. 
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Figure 3.1: Gluing two P 1 sheets into a hyperelliptic curve. 

For concretness we label the two P 1 -sheets as the upper and the lower sheet of X . The 
covering map v Q : X Q — > P 1 projects each sheet onto P 1 and the corresponding covering 
involution % Q : X Q — > X interchganges the two sheets. 

While working with loops on X Q (either as representatives of elements in 71*1 or as 
circles determining Dehn twists on X ) it will be convenient to describe these loops in terms 
of their zvimages m P 1 - We will only look at loops which do not pass through the branch 
points and which are transversal to the boundary circles of our sheets. Every such loop L 
projects via v Q onto a simple closed path in P 1 which does not pass through any branch 
point and is transversal to the branch cuts. Therefore specifying a loop L in X a is the same 
thing as specifying a simple closed path in P 1 (not passing through the branch points and 
transversal to the branch cuts), together with a labeling at each point indicating whether it 
is on the upper or lower sheet and such that this labeling changes upon crossing a branch 
cut. To avoid introducing additional notation we will write L both for the loop in X a and for 
the corresponding labeled path in P 1 . This will not create any confusion since it will always 
be clear from the context which incarnation of L we have in mind. 

The image of the geometric monodromy representation 

mon : n 1 (B,o) — > Map(X ) 
for the family / : X —>■ B, is the full hyperelliptic mapping class group 

A(X ) = {<pe MappQI^- 1 = *o}. 

It is generated by the right handed Dehn twists along the sequence of loops ai, . . . ,a 2g+ i 
depicted on Figure In this picture we use the convention that for paths in P 1 the solid 
pieces are on the upper sheet and the dotted pieces are on the lower sheet. 
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Figure 3.2: Dehn twists generating A(X ). 

Note that the above Dehn twists define a surjective homomorphism from the braid group 
B 2g+2 on 2(yf+2 strands to the hyperelliptic mapping class group A(X a ). Indeed, by definition 
B 2g+2 can be presented as 



B 2g + 2 — \ ti, t 2 , ■ ■ ■ , t 2g+ i 



titj = tjU, for \i — j\ > 2 

titi+lti tj_)_itjtj_)_i 



and so the assignment U \— > (Dehn twist along Oj) induces a (necessarily surjective) group 
homomorphism K g : B 2g+2 — > A(X ). 

Fix a positive odd integer n. To fix notation, choose a primitive n-th root of unity 7 G 
and let a : Z/n — * C x , k 1— > 7 fe be the corresponding character. For future use we denote 
the corresponding standard generators of S)iJ„ as follows 

a := (-1, 1,0, 1), a := (1, 1, 1,1), a := (1, 1,0, a). 

We also write e := (1, 1, 0, 1 ) for the identity element in DH n . 
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With this notation we are now ready to define the base representation p e Mb(X , n) at 
which we will be checking the open orbit condition (i) from section 12.21 For this we only 
need to exhibit a surjective homomorphism ip n : 7Ti(X ) — > DH n . We define ip n to be trivial 
on the complement of the branch cuts on both sheets and we postulate that the passing 
transformations Pj G DH n corresponding to going through the branch cut Cj should be: 

Pi = er, P 2 = acr, P3 = acr, P4 = a, P5 = ex, Pj = e, for all i > 6. 

Assume that g > 6, so there are at least two branch cuts with passing transformation equal 
to the identity. 

Consider next any two element orbit u = {x, X 1 } £ (Z/n x Z/n)/// 2 and let W u be the 
corresponding 2-dimensional irreducible representation of QH n . As we saw in section 12.21 
the action of DH n on W u factors through /x 2 K (Z/n x Z/n). Choose a basis {t> + ,t>_} of 
W u consisting of eigenvectors for the Z/n x Z/n action. To fix notation assume that v + 
corresponds to the character x an d that t>_ corresponds to the character x~ l - If we write 
the character x as X = {ct b ,a c ) for some integers b and c, then in the basis {t>+,t>_} the 
representation W u is given by associating 

<r (->• R, a^P b , a i-f P c , 

where P and P are the 2x2 matrices: 

== ".'1 , a -.-(° 1 



This gives the matrices for the action of passing transformations on the local system W u : 
(3.1) Pi = P, P 2 = P b R, Ps = P c R, P* = P b , P 5 = P C - 

and the rest are equal to the identity matrix I 6 GL 2 (C). 

Note that by our assumption on n it follows that P is of odd order so these matrices are 
never equal to — I. 

For any 1 < % < j < 2g + 2 let Ljj denote the loop which goes around the branch points % 
and j, passing under any other branch points which are in between on the real axis, and going 
in the clockwise direction. (Some sample loops Ly are illustrated on Figure l3~3l ) Assume 
that the lower part of the curve is on the upper sheet, and let e GL 2 (C) = GL(W„) 
denote the monodromy transformation around L^. 

Observe next that the representation W u is self-dual: in our basis {t>+, t>_} the invariant 
pairing Q : W u ® W u — » C is given by 



Q {{s)\s>)) :=rs+rs - 

The pairing Q induces an intersection pairing on Hi(X ,W u ) and an isomorphism of ho- 
mology and cohomology Hi(X ,W u ) = iP(X" ,W,J, both compatible with the monodromy 
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Figure 3.3: Some L^-'s. 

action of the hyperelliptic mapping class group tti(B,o) = A(X ). Thus it suffices to calcu- 
late the monodromy action on the homology Hi(X ,W u ). 

We will represent the elements in Hi(X ,W u ) by loop-like chains. Similarly to ordinary 
loops, a looplike chain with coefficients in W u is given by: 

• an oriented simple closed path in P 1 (not passing through the branch points and 
transversal to the branch cut), together with a labeling at each point indicating whether 
it is on the upper or the lower sheet, and such that the labeling changes upon crossing 
of a branch cut; 

• a specification at each point of the path of a vector in W u , such that upon crossing 
a branch cut C, from the upper to the lower sheet this vector is modified by the 
corresponding passing matrix Pj. 
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A looplike chain will typically be denoted by vL, where L is the loop and v is the corre- 
sponding vector in W u . 

Our convention for the intersection pairing on Hi(X ,W u ) will be that we intersect the 
looplike chains by intersecting the underlying loops according to the right-hand rule (a first 
path intersects positively a second path which, to him is coming from the right) and at each 
intersection point we pair the corresponding elements in W u via Q. 

The homology group Hi(X Q , W u ) contains elements of the form VijLij, where G W u is 
any vector which is invariant under M^. We can distinguish three cases: 

if My is the identity then can be any vector so there is a two-dimensional space of 
such cycles; 

if Mij is a reflection (of the form P k R) then is of the form [v + M^v) for any 
sufficiently general vector in W u , and in fact we may take v = v + ; 

<0 if is a rotation of the form P k for k different from modulo the order of P, then 
there are no nonzero cycles of this form. 

Using these elements we can now prove the following: 

Lemma 3.1 The cycles of the form v^Lij span the homology Hi(X ,W u ) of the hyperelliptic 
curve with coefficients in W u . 

Proof. We need to separate into cases depending on b and c. Assume first of all that b and c 
are different and different from (modulo n) . For the purposes of this lemma, we can apply 
a braid transformation to arrange things so that the passing matrices are (in order starting 
with Pi): 

P b , P c , P h R, P C R, R, I,.... 

Now we consider an element of the homology of the hyperelliptic curve with coefficients in 
W„. It can be moved to a cycle supported over the real axis, necessarily on the interval 
between 1 and 2g + 2. Look first at the interval [2g + 1, 2g + 2]. By subtracting off a cycle 
of the form t>23+i, 23+2-^23+1, 23+2 we obtain a cycle which is zero on the upper sheet along 
the interval in question (note that the vector v 2g+i, 23+2 can be arbitrary). Now the cycle 
condition at the point 2g + 2 implies that the cycle is also zero on the lower sheet. We get 
to a cycle supported on the interval [1, 2g + 1]. Continuing this way by induction we get to 
a cycle supported on the interval [1, 10] (the last nontrivial passing matrix is P5). 

The monodromy transformation Mg^o is the identity and so again we can take fg^o to 
be an arbitrary vector. By subtracting a multiple of ^9,10^9,10 we get to a cycle supported 
in [1,9]. 

Next look at the monodromy matrices 

M 2 , 9 = RP b and M 4 , 9 = RP C . 
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Thus we get that up to scalars 

v 2 , 9 = (1 + RP h )v + = 

and 

v^ = (l + RP c )v + = 

The determinant of the matrix with these two vectors in the columns is 7 C — 7 b which is 
nonzero under our assumption that b is different from c. Therefore by subtracting off an 
appropriate combination of the cycles ^2,9-^2,9 and v^qL^q we obtain a cycle which is zero 
in the interval [8,9], in other words it is supported on [1,8]. Again the monodromy matrix 
M 7 8 is the identity so we can subtract off a vector of the form t> 7j8 to get a cycle supported 
in [1,7]. 

We repeat the argument above using 

v 2J = (1 + P b RP b ) v+ = v+, and 
V4J = (1 + P b RP c )v + = v + + 7 C ~V 

which are linearly independent. This, combined with subtracting off a 1*5,6-^5,6, gets us to a 
cycle supported in [1,5]; and repeating again the same argument with 1*2,5 and 1*4,5 we get 
to a cycle supported in [1,3]. On the other hand the monodromy transformation M 12 is 
trivial so by subtracting off a cycle of the form v 1,2-^1,2 with v i j2 arbitrary, we get to a cycle 
supported in [2,3], which must be a multiple of t* 2,3-^2,3 so we are done. One can note in 
passing that this last cycle must automatically be zero since the monodromy M 2j 3 doesn't 
have any fixed vectors but this is not really important. This completes the proof in the case 
b^c. 

Suppose now we are in the case b = c (modulo n). Then b and c are nonzero. Thus we 
can repeat the same argument as above but arranging things so that the passing matrices 
are (in order) 

P\ P c , I, P b R, P C R, R, J, . . . . 

In this case the same argument as before (but using vectors such as f 4j n and 1*6,11 etc.) 
allows us to get to a cycle supported on [1,5]. Now the first monodromy matrices are the 
identities: 

Mi, 2 = M 2 , 3 = M 3j4 = I 

so we can subtract off cycles of the form t>i j2 £i,2) ^2,3^2,3, and 1*3,4-^3,4 to get to a cycle 
supported on [4, 5] and again we are done. This completes the proof of the lemma. □ 

For the remainder of the argument we return to the labeling (|3.1|) for the order of the branch 
cuts. 

Now let tij G Map(X D ) denote the right handed Dehn twist along the loop L^. For the 
three types of behavior of My we have: 
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if Mij is the identity, then ty e A[X Q ) and acts on the local system i? 1 / ;t W M . In 
particular £y maps to a well defined element Dij e Sp(-£/i(X D , W u )); 

if My is a reflection, then £?■ e A(X„) and acts on the local system R l f*W u . In par- 
ticular £?• maps to a well defined element Df- e Sp(ifi(X , W u )) (Figure EH illustrates 
the typical action of Df-)] 

if Mi~ is a rotation then we don't consider the Dehn twist. 




Figure 3.4: The action of D| 3 on the loop L34. 



For uniformity we will always consider Df-. We now have the following lemma: 
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Lemma 3.2 The subgroup of Sp(Hi(X ,W u )) generated by the elements 

1 < i < j < 2# + 2l 
Mij ^ rotation J 

acts irreducibly on the complex vector space Hi(X ,W u ) . 

Proof. Consider the elements of the the form Df^ — 1 in the group algebra of Sp(Hi(X , W u )). 
By general principles this algebra is semisimple, so it suffices to find a vector wq such that 
the subspace generated by the action of the Df- — 1 starting with w , spans the whole 
#i(X ,W u ). We will start with 

w := ^2,11-^2,11 

and show that using the Dfj — 1 we can get to any vector VijLij. In view of Lemma f3. II this 
will complete the proof of the irreducibility. 

Note that P 6 is the identity and Pi is a reflection, so M 2i n = R is a reflection. Thus v 2) ii 
is of the form v + + Rv + = v + + v _ . 

Using D\ 2 — 1 we get to {v + + Rv + )L\ 2 (note that we allow ourselves to multiply by a 
factor for example ^ or — | when we say this). 

Now one of b or c is different from zero modulo the order of P. Assume for example that 
b is different from zero. Then the monodromy transformation M 2 j is P~ b R so using D 27 — 1 
we get to 

(l + p- b R)(l + R)v + L 27 . 
Applying again D\ 2 — 1 we get back to 

(l + p- b R)(l + R)v+L l2 . 

In the other case where b is zero but c nonzero we could use D\ 9 — 1 and get to 

(1 + P- C R)(1 + R)v + L 12 . 

In the first case, note that the image of (1 + R) is not contained in the kernel of (1 + P~ b R), 
and the image of (1 + P~ b R) is linearly independent from the image of (1 + R), so with the 
vector we obtained previously we obtain both vectors v + Li 2 and v~L\ 2 . The same holds in 
the second case where we used the matrix P c . 

A similar argument gets us to any of the vectors fL 34 and vL 56 . 

Next, using again the fact that one of the P b R or P C R is different from R, and using 
the appropriate transformation D\ n — 1 or D\ u — 1 as well as D\ n — 1 and following by 
D\ x 12 — 1, we get to any vector of the form fL 1112 . Now using the Dehn twists for i, j with 
11 < i < j < 2g + 2 we obtain all of the vectors of the form vLij for 11 < i < j < 2g. 

From these using the Dehn twists Df - — 1 for i < 11 and j > 11 we get to all vectors of 
the form v^Lij for i < 11 and j > 11. 

Similarly we get to all vectors of the form v^Lij when the monodromy transformations 
are reflections, by using the Dehn twist Df, — 1 on a vector VkiL ki where one of k or / 
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is either i or j and where [k, 1} is a branch cut on which the passing matrix is a reflection 
(note that such k, I always exist when the monodromy is a reflection). Here from above 
we have already gotten to the VkiL^i with Vki arbitrary 

This argument also works to obtain VijLij whenever 1 < i < j < 6 is an exceptional case 
where the monodromy is the identity due to a special equality of the form b = c or b = or 
c = 0. 

The only cycles which remain to be obtained are the v^Lij for 7<z<j<10. We first 
obtain v-j%L-j%. To do this, note that two among the three matrices R, P b R and P C R are 
different, and for appropriate choices of i and j corresponding to these two, chosen among 
2, 4 and 6, we have that the images of the rank one matrices (1 + Mij) and (1 + Mjj) are 
linearly independent and span our two dimensional space (this is similar to the argument 
used in Lemma EHJ). Thus applying the Dehn twist Dj 8 — 1 to the vectors Vij and vjj 
we span a two dimensional space so we can get to any vector of the form v 78-^73 with v-js 
arbitrary. The same argument yields any vector of the form ^9,10-^9,10 with Ug,io arbitrary. 
Finally, in the exceptional case where M§g is the identity (this is when b = c) , using its Dehn 
twist we get to the vectors of the form v 89 L 89 . 

This completes the proof of the lemma. □ 

Let now g denote the Lie algebra of the monodromy group acting on the representation 
C 49-4 = H 1 (X ,W U ). Note that g is semisimple, by general theory. In Lemma 13.21 we 
proved that H l (X Q , W u ) is an irreducible representation of g. The above proof works for the 
Lie algebra since the Dfj are unipotent matrices with Jordan blocks of length at most one 
and thus Aij := Df- — 1 are elements of g. Hence the above proof shows that the act 
irreducibly. 

For some i,j we have decomposing into two Jordan blocks of length one (this is the 
case for A\^ ^3,4 etc.). However there are some i,j where the monodromy Mjj is a reflection 
(for example i — 2, j — 11), where the Ajj has a single Jordan block of length one. 

Next, note that by isolating the monodromy representation on the part of the curve where 
the local system is trivial, we obtain a monodromy representation of the direct sum of two 
copies of the cohomology of a hyperelliptic curve of genus g' with the monodromy acting 
diagonally. If we take all branch points for i > 11 then this has genus g' = g — 5. Deligne's 
argument from |Deligne80| Section 4.4] or the argument from |Janssen83] . works for the 
hyperelliptic monodromy action on the standard cohomology (the monodromy is generated 
by conjugate Dehn twists) so this monodromy group is Sp(2g — 10). In particular we have 

sp{2g - 10) C g C sp(4g - 4) 

where the composite inclusion is the linear embedding of the diagonal action on the direct 
sum of two copies of the standard representation of sp(2g — 10). 

This shows that our monodromy action satisfies the hypothesis of the following purely 
algebraic theorem: 
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Theorem 3.3 There exists go with the following property: suppose g > go and suppose g is 
a semisimple Lie algebra sitting in a pair of inclusions 

sp(2g-10) CgCsp(%-4) 

where the composite inclusion is the linear embedding of the diagonal action on the direct 
sum of two copies of the standard representation of sp(2g — 10). Suppose that the action of 
g on C 49-4 is irreducible, and suppose g contains an element A which acts on C 4s ~ 4 with a 
single Jordan block of length one. Then g = sp(4g — 4). 

Proof. We first claim that g is simple. If that were not the case, then we could write 
9 = Bi x 02 and so the representation W = C 49-4 would decompose as an exterior tensor 
product = Wi W% of representations of gi and 02- This however can be ruled out by 
looking at the element A. If it is nontrivial in both factors then it would act on W by a 
Jordan normal form which is the tensor product of two nontrivial Jordan normal forms, in 
particular it would have a Jordan block of length > 1; if it was nontrivial in only one of 
the factors then it would act by the tensor product of a nontrivial Jordan form, by a trivial 
vector space (of dimension > 1); thus it would have at least two Jordan blocks. In either 
case this contradicts the hypothesis that A acts on C 4s ~ 4 with a single Jordan block of length 
one. This proves that g is simple. 

Now we can choose go big enough so that the dimension of sp(2g — 10) is bigger than 
the dimension of the exceptional simple Lie algebras. By classification, this means that g is 
of one of so (2m), so(2m — 1), sp(2m), or sl(m). Looking at dimensions, we get m > g — C 
where C is some constant. 

On the other hand, note that all the fundamental representations of the classical groups 
are essentially (the only exception being the spin representation) the wedge powers of the 
standard representation. Combined with Weyl's dimension formula this implies that there 
is an mo such that for m > mo the only irreducible representations of dimension < 5m of 
one of the classical groups above, are the fundamental representation or (in the last case) 
the dual of the fundamental representation. 

This claim gives that g acts by the standard representation, which immediately implies 
that it is equal to sp(4g — 4) (it can't be orthogonal or special linear because we already 
know it is contained in the symplectic group). □ 

We are now in a position to complete the 

Proof of Theorem [XJ Let go be such that Theorem 13.31 holds. In view of Corollary 12.41 
and Lemma [2.81 we only need to show that the Zariski closure G of the monodromy action 
of n 1 (B, o) on H\(X , ad(p)) acts with an open orbit on Hi(X , ad(p)). 

By Theorem 13.31 we get that each subspace Hi(X ,W u ) C Hi(X Q , ad(p)) yields a mon- 
odromy representation equal to the symplectic group. Note that Theorem 13.31 implies that 
the Lie algebra of the monodromy group is equal to sp(4g — 4). However, in view of the fact 
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that we already have an inclusion of the monodromy group of Hi(X , W M ) in Sp(4g — 4), we 
get that this inclusion is surjective so the monodromy group is equal to Sp(4g — 4) which we 
now write as Sp(Hi(X Q , W u )). 

In particular we have a natural inclusion 

GcSp(iJ 1 (X ,C))x J] Sp^pT^Wj), 

so that the projection on each factor is surjective. On the other hand, G is semisimple. 
Going back to the level of Lie algebras, this implies that the simple summands of q are 
sp(2g) occuring once, and sp(Hi(X 0) W u )) = sp(4g — 4) occuring a certain number of times. 
Call these summands Si, . . . ,Sk- 

Each irreducible factor Hi(X , W u ) in the representation H 1 (X Q , ad(p)), is an irreducible 
representation of the Lie algebra 

sp(2g) 0Si © ... ©fifc. 

As such, it decomposes a priori into an exterior tensor product of representations of the 
summands; but by dimension considerations, this tensor product must just be an irreducible 
representation of one of the summands Sj. Also this representation is isomorphic to the 
standard representation of Si = sp(4g — 4). Thus each Hi(X 0) W u ) comes from the standard 
representation composed with a projection onto one of the factors. Note also that the 
representation C 2g comes from the standard representation composed with the projection 
onto the factor sp(2g). 

The above statements on the level of Lie algebras imply the same things for the connected 
components of the Lie groups. We obtain that the connected component of the monodromy 
group G decomposes as a product 

G° = Sp{2g) x Si x ...xS k 

where each Si is equal to Sp(4g — 4) and Si acts on a direct sum of r\ copies of its standard 
representation. 

Our goal now is to prove that all of the r« are equal to 1. Suppose the contrary, i.e. 
suppose that there are two distinct components u and v! such that the Sciihg S{ ctcts on 
#i(X ,W u ) and #ipT o ,W u 0- In particular this means that #ipT ,W u ) and #i(X ,W u ,) 
are isomorphic as representations of G°. 

The elements Dfj are unipotent so they go into G°. Let T denote the subgroup of the 
monodromy group which maps into G°. We obtain a map 

C[r] -^C[G ]. 

In particular the action of the group algebra C[r] on u iJ 1 (X o ,W u ) factors through the 
action of the group algebra of G°. Thus, with our assumption of the previous paragraph 
that some r\ is > 1, we would get two components Hi(X ,W u ) and Hi(X ,W u ') which 
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are isomorphic as representations of the group algebra C[r]. Therefore for every element 
E e C[r], the eigenvalues of E acting on Hi(X ,W u ) and Hi(X ,W u >) are the same. We 
will write down elements E which act with single nonzero eigenvalues; thus these values are 
the same for u and u' . We will show that this implies that u and u' are the same component. 
This will contradict the assumption that n > 1 so it will prove the statement that all of the 
r\ are equal to 1. 

We will write down our E as products of elements of the form := Df- — 1. The 
component u (respectively u') is determined by the numbers b and c (respectively b' and 
c') which occur above. These are taken modulo the order n of the root of unity 7, and 
interchanging b <-> —b and c <-> — c doesn't change the dihedral component u. Thus, proving 
that the two components are the same means that we want to show 

(b,c) = ±{b',c') in {Z/nf. 

Let i,j = 7, 9, 11. Look for example at 

A 2 ,iA h2 . 

This takes the vector v 2 jL 2 j first to v 2 jLi t2 and then to 

(1 + P •'•'/.')/•,.,/,„ 
where = 6, c or depending on whether i — 7, 9 or 11. Look now at 

E = A 2:i A 1:2 A 2:j A 1:2 . 

It has an image vector which is a multiple of v 2 ^L 2 ^ so this can be its only nonzero eigenvector. 
Its action on this vector is 



Ev 2ti L 2ji = (1 + P~ Xi R)(l + /' W,\)r,.,/. 2 . ( . 

Note also that the matrix (l+P~ Xi R) is itself of rank one so the product of matrices appearing 
above also has a single nonzero eigenvalue. In particular the unique nonzero eigenvalue of 
B is equal to the unique nonzero eigenvalue of the matrix (1 + P~ Xl R)(l + P~ x iR). This 
matrix may be written as 

(\ -|- r^ x j~ x i ry— x i _|_ ry~ x j \ 
ry X i _|_ ry X 3 \ _|_ r^ x i^ x j J 

Its eigenvector is 

^ = (i+p-"%= ( 7 i 4 ). 

Calculating 

(]_ _|_ sy x j~ x i sy~ x i _)_ sy~ x j \ / ]_ \ / 2 + ry x 3~ x i -)- sy x i~ x j \ 

r^ x i _|_ /y^' ]_ _|_ r^ x i — ' x i J \ r^ x i J \ 2,^' -\- -\- sy^Xi—Xj J 
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says that the eigenvalue is equal to 

2 -)- ry X i~ X i _|_ ry X i~ X j _ 

Now take various values of i and j, and compare the results for the components u and 
v! . We obtain: 



from z = 11, j = 7, 
from z = 11, j = 9, 
and from % = 9,j '• = 7, 



7+7 =7+7 5 



7+7 =7+7 



7 +7 =7 +7 



In Z/n these equations give: 

b = ±b', c = ±c', (b-c) = +(&'- c'). 
The first two equations admit four possibilities: either 

(6,c) = (6 / ,c / ), or (b,c) = -(b',c'), 

or else 

(6,c) = ±(6',-c / ). 

The first two possibilities are what we want to show. In the last two possibilities we have 

b-c= ±{b' + c'). 

Thus the third equation above says 

b' + c' = ±(b'-c'). 

This says that either d = —d or else b' = —b'. In either case, the equation (6, c) = +(&', —d) 
gets transformed into the equation (6, c) = ±(6',c') so we are done. In fact, we could have 
noted here that since n is odd, the equations d = —d or b' = —b' do not occur, but the 
present argument works even when n is even. 

We have now shown that if Hi(X ,W u ) and Hi(X ,W u >) are isomorphic as representa- 
tions of G° then u and v! represent the same dihedral component of our representation. This 
implies that all of the rj are equal to one, which in turn gives that 

G° = Sp(2(7)x J] Sp(#i(X ,W u )). 

i^(l,Q) 

(Actually, the same is true of the full monodromy group because the full group is also con- 
tained in this product of symplectic groups: G = G°.) This group acts on its representation 
C 2fl ©©^(j o) Hi(X Q ,W u ) with an open orbit. Combined with Lemma and Theorem 12. 21 
this completes the proof of Theorem EI 1=1 
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3.2 Lefschetz pencils 



We would now like to extend the techniques of the previous section in order to prove Theo- 
rem El 

Let Z be a smooth projective surface with b\(Z) = 0. Let 0z(l) be a very ample line 
bundle on Z and let P 1 C F(H°(Z, Oz(k))) be a generic line. Denote by e : Z — > Z the 
blow-up of Z at the base points of the pencil of curves {D t } t& i, and let / : Z — > P 1 be 
the corresponding Lefschetz fibration. Let pi, . . . ,Pn G P 1 be the critical points of / and 
let B = P 1 - {px, . . . and X = f^ 1 {B). Let X t , t G B denote the fiber of / over t, or 
equivalently, the strict transform of the divisor D t C Z. 

Fix g so that Theorem 13.31 applies. Then, as we saw at the end of the previous sec- 
tion, it follows that the monodromy group of the hyperelliptic family of genus go acting 
on the cohomology of the full local system End(V n ) corresponding to ad(p) is equal to 
Sp(2# ) x n u Sp(Fi(X ,W u )) with Sp(#i(X , W„)) = Sp(4# - 4). We also assume that 
go > 6, so there are several branch cuts along which the representation p is the identity. Set 
m := 25-0 + 1. 

Let £ := Oz{k) denote our line bundle. Our assertions will be made for k big enough. 
Let D C FH°(Z, £) denote the discriminant locus consisting of the sections defining singular 
curves. We will fix a base point o G FH°(Z, £) — D (chosen specially below); and as always 
X a will denote the smooth curve defined by the section o. Then 7ri(P(if°) — B, o) acts by 
diffeomorphisms on X Q and hence it acts on Mb(X ,ti). Furthermore, by the Lefschetz 
hyperplane section theorem, the geometric monodromy action 7Ti(B,o) — *> Map(X D ) for the 
family / : X — ► B factors through the natural map tti(B,o) — > tti(F(H°) — D, o) and so it 
suffices to show that tti(F(H°) — D, 6) acts on Ms(X Q ,n) with a Zariski dense orbit. 

As before we shall fix a local system p on X a , with finite monodromy factoring through 
a representation of the dihedral Heisenberg group. Then there is a subgroup of finite index 
in 71"! (Pi/ — D, o) which preserves p, so it acts on the space 



According to Corollary 12.41 and Lemma 12.81 we only need to show that the Zariski closure 
G of the image of this monodromy action acts with an open orbit on H l (X OJ ad(p)). Our 
technique will be to show that G is as big as possible, given the above decomposition and the 
fact that it preserves symplectic forms on everything. To achieve this we will use a family of 
curves in the linear system \£\ which have hyperelliptic handles and will apply to the curves 
the results for the hyperelliptic case obtained in Section 13.11 

We will define a particular subspace E C FH°(Z, £), and among other things choose 
o G E-DflE. Then the fundamental group 7Ti(E— DDE, 6) is contained in 7^i(H°— D, o) and 
again a finite index subgroup will act on f/" 1 (X G , ad(p)). The subspace E will be designed so 
that 7ii(E — D n E, o) preserves the handle decomposition of X Q . Using this we will obtain 
first a smaller subgroup of G and then apply an argument using loops in the full space 
FH°(Z, £) — D (which don't preserve the handle decomposition) to obtain the full group G. 



T [p] M B (X ,n) = H\X , a d(p)) = H\X ,C) © H\X ,W U ) 
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Fix a point P G Z. Choose a collection of sections a , {sjKLo G H°(Z, £) so that 

for a suitable local trivialization of £ and local coordinates (x, y) near P we have: 

(a) near P we have ao = x m — y 2 ; 

(b) the section ao has no singularities other than P; 

(c) near P the sections have the form a;*; 

(d) the sections Sj vanish to order at least m + 2 at P and together with ao form a basis 

for a linear system which has no base points outside of P. 

It is clear that by taking k big enough we can always find such ao, fi and Sj. In addition we 
will need to choose the Sj's so that they satisfy certain connectedness conditions which we 
shall describe further on ( Lemma 13.41 and Lemma [3. 8|) . 

With these choices we let E be the affine space of sections of £ of the form 

m-l N 

a + tifi + ZjSj, 

i=0 j=0 

and we take as a base point the point o G E corresponding to the values Zj = and i« = 
for z > 0, with £ — 1- Write this as o = (1, 0, . . . , 0). We will work on open polydisks in 
E of the form \to — 1| < B, \ti\ < B for i > and \zj\ < C for all j (with S and C to be 
determined later), possibly after rescaling the fi. 

To achieve the desired behavior of the monodromy we start by looking at the choice of the 
Sj and C. For k big enough we can choose a linearly independent family of sections Sj such 
that the Sj all vanish to order > m at P, and such that the linear system they generate is 
without base points away from P. Let G C FH°(X, £) denote the subspace generated by the 
Sj and by a . It is a projective space, with a codimension one projective subspace Goo C G 
corresponding to the linear system spanned by the Sj without ao- The family of sections 
a + Ylj z j s j provides a system of affine coordinates Zj for the complementary affine space 
Aq := G — Gqq. On the other hand, over Aq the universal family of curves is a family which 
is holomorphically locally trivial near the singular point P. Indeed, over any small enough 
disc in the coordinates Zj, one can choose local coordinates at P, depending on the z/s, such 
that a + J2j z j s j has the form x(zj) m — y(zj) 2 . Let B<g denote the subset of points in G 
parametrizing curves that have singularities outside of P, union the G^ which corresponds 
to curves with bigger singularities than usual at P. Over G — B>q we obtain a family of curves 
which are smooth except for their singularities at P, and the family is holomorphically locally 
trivial (hence topologically locally trivial) along the section corresponding to the point P. 

Let s' be a point in the complement G — ©g and let X s > be the fiber over s'. With m 
odd, the singularity of each fiber at P is a higher-order cusp, so in fact the fibers such as 
Xgi are singular curves which are topologically (but not differentially) manifolds. The local 
topological triviality of the family means that it makes sense to speak of the monodromy 
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action of tti(G — U>&, s') on the cohomology of the fiber X s i with coefficients in the trivial 
local system. 

Now we come to the first of the connectedness conditions referred to above. In fact 
the subject of the monodromy of pencils having singularities especially at the base locus, 
has been intensively studied recently notably by Tibar |Tibar02a| |Tibar02b . Our situation 
above is a very special easy case of this phenomenon so we don't need to call upon his general 
results. 

Lemma 3.4 For k big enough and by choosing a big enough family of sections Sj, we can 
ensure that the Zariski closure of the monodromy action 0/711 (G — Cg, s') on H (X s i, C) is 
the full symplectic group. 

Proof. If we choose a general line A 1 C Aq then by standard Lefschetz theory (see e.g. 
Katz68 j5 |Looijenga 96j) the fundamental group of G — D G is generated by the loops in 
A 1 — A 1 fl Be- Here (and this is the important point of the argument) we can take only 
the loops which go around the points in A 1 fl H>&; we don't need to look at the loop going 
around the point at infinity since it is the product of the others. Thus the fundamental 
group of G — ©g is generated by loops going around the affine part of the discriminant 
D G - Goo = D G n A G . 

For k big enough and by choosing a big enough family of sections Sj, the affine part of 
the discriminant divisor D G — Goo is irreducible, hence connected. Note that we could never 
assure that D G is connected since it contains Goo as an irreducible component — thus the 
importance of saying that the monodromy is generated by loops around the affine piece. To 
get this connectedness we follow the standard argument in the theory of Lefschetz pencils: 
the discriminant divisor in the affine piece is the image of an affine space bundle over the 
surface Z — P. For big enough values of k this family of affine spaces (which to a point 
x G Z — P associates the subspace of sections in A& which are singular at x) is a vector 
bundle over Z — P; thus its image is irreducible. Of course we also choose k so that the 
general point in this divisor corresponds to an ordinary double point of the curve. 

Now the Kazhdan-Margulis result as reported by Deligne |Deligne80 5.10] works the same 



way to show that the monodromy of the fundamental group of G — H)& on the cohomology of 
the family of curves with trivial coefficients, has Zariski closure equal to the full symplectic 
group. Indeed the monodromy around a point of B<g — Goo is a symplectic transvection 
(because the singular curve has an ordinary node), and the connectedness of the divisor 
means that all of these elements are conjugate. As we have seen above, they generate 
the monodromy group, so we have a group generated by a family of conjugate symplectic 
transvections. Furthermore the monodromy representation has no fixed vectors (a fixed 
vector would correspond to a class in H 1 {Z ) C) which we have assumed is trivial). □ 

Choose k and the Sj as per the above lemma. Choose an explicit collection of loops 7^ in 
G — ©g which generate the monodromy, and choose C big enough so that these loops are 
contained in the region \zj\ < C. 
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Our coordinate patch around P will consist of a nested pair of balls U C U' C Z together 
with a pair of coordinate functions (x, y) : U' — > C 2 sending [/ (respectively U') to the ball 
of radius T (respectively T') in C 2 . We can assume that x and y come from sections of 
Oz{ko) for some ko, via a trivialization of this last line bundle over U' . Furthermore the 
trivialization can be assumed to come from a given section u of Cz(l) which doesn't vanish 
at P. Then provisionally put 

/. : = x^-^ for < i < m - 1 and a := x m u k - mk ° - y 2 u k - 2k °. 

These come from global sections of O z (k) (choose k > mk ). 

Lemma 3.5 For any fixed choice of the sections Sj and a , of the constants T and T' , and 
for any 5 > 0, we can make a rescaling of our local coordinates and of the so that the 
following hold: 



• we can retain the properties (a) and (c), while Oq and the Sj remain fixed in H (Z, C); 



■- , I ■ ill u in /(,( <:.<< in i I " ' 

• the sections fa become arbitrarily small inside H°(Z,C); 

• the coordinate patches U and U' become arbitrarily small inside Z , and 



E 



z j s j 



< 5 



on the coordinate patch U' , for all \zj\ < C . 

Proof. We can rescale a; by a factor of A 2 and y by a factor of A m , and the trivialization by 
a factor of A~ 2m (in other words scale u by a factor of A 2m / fco ). We retain the expressions 
(a) and (c), a remains fixed in H°(Z, C), and the sections /j become arbitrarily small inside 
H°(Z,C). Also the coordinate patches U and U' become arbitrarily small inside Z. Finally, 
since the Sj (which are fixed) vanish to order > m at P, for any 5 > we can choose the 
rescaling so that the required estimate holds. □ 



The family of curves of the form 

m—1 m—1 
i=0 i=0 

gives the full family of hyperelliptic curves in our coordinate patch. In particular the mon- 
odromy of this family (i.e. the fundamental group of the complement of the discriminant 
locus) acts as the braid group of braids on m strands. Choose loops generating this mon- 
odromy and choose B so that the loops are contained in the region \U\ < B. 
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Let dll be the spherical boundary of the coordinate patch. In terms of the local co- 
ordinates the equation for dll is | (x, y)\ = T. For T big enough, the intersection of the 
curve 

m— 1 
i=0 

with the sphere dU will remain approximately equal to a single fixed circle as £; vary in the 
region \ti\ < B. As pointed out above, if we made a sufficient rescaling at the start then the 
ball U will still be small in Z. Furthermore, by the estimate of Lemma 13 .b\ addition of the 
terms ^ . ZjSj will not move the intersection circle by very much either. 

We have thus found parameters for our family of curves E (and a rescaling of the fi) 
such that for any point (t, z) in the family satisfying the bounds < B and \zj\ < C, the 
intersection Xn z ) D dU remains close to a single fixed circle. 

Finally we choose as a base point o — (1, 0, . . . , 0) i.e. t = 1. By choosing 6 small enough 
in the above choices, we can insure using the estimate of Lemma [3.51 that when we let the z 
coordinates go around the loops 7*., the piece of the curve inside U doesn't move too far from 
the base curve and in particular the monodromy action on this piece is trivial. Similarly, 
note that with a very small scaling factor A the sections fi become very small compared to 
a (or more precisely, they become small compared to the differential of ao along the zero-set 
of ao). So when t goes around loops generating the braid group action, this doesn't move 
very much the curve outside of U' . Furthermore we choose U so that these loops act trivially 
in the coordinate patch on U' — U . Thus when t moves we obtain a braid group action on 
the piece of the curve inside U and a trivial action on the piece outside U. We can sum all 
this up as follows: 

Corollary 3.6 Over E the family of curves decomposes into a family of hyperelliptic curves 
of genus g joined onto a family of curves of genus g — g along a circle which stays essentially 
fixed. There is a collection of paths in E which generate a monodromy action on the genus 
g — go piece whose Zariski closure is the full symplectic group of the cohomology of the genus 
g — go curves (Lemma \3.J$ . These paths act trivially on the hyperelliptic piece. On the other 
hand, there are paths in E generating the braid group action on the hyperelliptic piece, which 
in turn act trivially on the piece of genus g — go- 



Next, fix our base representation p to be trivial on the genus g — go piece and equal to 
the dihedral Heisenberg representation chosen in the hyperelliptic argument of the previous 
sections for the hyperelliptic genus go piece. Note that since m = 2g + 1 rather than 2g + 2, 
one of the branch points on the hyperelliptic handle is at infinity; we assume that this branch 
point is part of a branch cut on which the passing matrix is trivial. Furthermore there is at 
least one other branch cut on which the passing matrix is trivial too. 

If we consider monodromy elements coming from the family E, these preserve the cutting- 
up of our curve into pieces X C\U of genus go and X Q — X Q nU of genus g—go- This monodromy 
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group preserves the decomposition of the cohomology of ad(p) into a sum of two pieces, one 
of dimension 2n 2 (g — g ) and the other of dimension 



2(n 2 -l)(< ?0 -l) + 2< 7o . 



The first piece corresponds to the monodromy on the sum of n 2 copies of the trivial rep- 
resentation, since p and thus ad(p) are trivial outside X Q n U. Now the cohomology of 
X Q — X Q fl U with coefficients in the trivial local system is isomorphic to the cohomology of 
the Riemann surface obtained by glueing in a disc along the boundary X Q fl dU. In turn, this 
Riemann surface is homeomorphic to the cusp curve considered in the family parametrized 
by G above. Thus the monodromy result of Lemma f3 .41 implies that the monodromy action 
of the loops 7^ on the cohomology of X Q — X C\U with coefficients in the trivial local system, 
has Zariski closure equal to the symplectic group Sp(2(g — go)). Now taking into account 
the fact that the restriction of V n to X Q — X a fl U is the direct sum of n 2 copies of the trivial 
representation; we get that the monodromy action on this first piece is a diagonal copy of 
Sp(2(g — go)) embedded in the product of n 2 copies of its standard representation. 

The second piece corresponds to the monodromy action on the cohomology of our genus 
go handle with coefficients in ad(p). We have an action of the braid group B m on this second 
piece equal to the braid monodromy action for the family of hyperelliptic curves. Of course 
not all elements of B m preserve the representation p, and as before we look only at elements 
which preserve p. The results of Section 13.11 apply here, giving the Zariski closure of the 
monodromy of this braid action on the cohomology of the hyperelliptic piece. 

We are now in the following situation. Consider the cohomology ^(Xg, ad(p)). Let 
i7 1 (X D , ad(p)) hyper and i7 1 (X , a.d(p)) 9 ~ 90 respectively denote the cohomologies of ad(p) over 
the hyperelliptic piece and the complementary piece. Note that in if 1 (X D , ad(p)) hyper we 
restrict to classes which are zero on the boundary circle joining the two pieces. By Mayer- 
Vietoris, restriction gives an isomorphism 



Furthermore both pieces have natural symplectic forms and the symplectic form on 
iJ 1 (X D , ad(p)) corresponds to the direct sum. 
Combined with the decomposition 



H\X , ad(p)) ^ H\X , ad(p)) hyper © H\X , ad(p)) 




this yields splittings 



H\X Q , ad(p)) hypcr = H\X , C) hypcr © H\X , W u ) 




hyper 
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and similarly 



H 1 (x o M{p)) a - a0 = H 1 (x o ,C)°-*>®l H\x ,w u y-°° 

Again these decompositions are compatible with the symplectic form. The action of 
7Ti(E — E n D, o) on H l (X , ad(p)) preserves this decomposition. 

Let G denote the global monodromy group i.e. the complex Zariski closure of the mon- 
odromy image of tti(¥H°(Z, C) — D, 6) acting on ad(p)). Let G hyper (respectively 

G 9 ~ 90 ) denote the Zariski closures of the monodromy of the family E (i.e. of the image of 
7Ti(E — E fl D, o)) acting on each of the pieces in the above decomposition. 

Lemma 3.7 With these notations, the product group is contained in the global monodromy: 

G hypcr x G g-g q 

Furthermore, 

G h ^ = Sp(2g )x J] Sp(H\X ,W u ) h ™ e *), 

and G 9 ~ 90 is the diagonal copy of Sp(2(# - g )) acting on H l (X X) 9 ~ 90 = C 2n2(9 " 3o) . 

Proof. The loops discussed in Corollary 13.61 generate a subgroup of G which factors as a 
product of subgroups of G hyper and G 9 ~ 90 , since the loops preserve the decomposition along 
dU and act trivially on one side or the other. The subgroup generated by these loops is also 
a subgroup of G hyper x G 9 ~ 90 . However, these loops generate inside G 9 ~ 90 the diagonal copy 
of Sp(2(# - g )) acting on H\X , C) 9 ~ 90 = C 2n2 ^ 9 ~ 9 ^ (cf the discussion above), and for the 
hyperelliptic case by the result of the previous section, the group 

Sp(2so)x H Sp(if 1 (l 0l W tl ) hwcr )cG hj ' per . 

w^(l ,Q) 

For general reasons, neither G 9 ~ 90 nor G hyper can be any bigger than these subgroups gen- 
erated by our loops. Therefore our loops generate all of G 9 ~ 90 (respectively G hjper ), and we 
obtain that 

ghyper x G g-g Q q 

The lemma is proven. □ 



To finish the proof of Theorem El we need some elements of G which mix up the factors 
H 1 (X , ad(p)) hypcr and if 1 (X , &d(p)) 9 ~ 90 . We get these by the following an "interchange of 
singularities" argument, which basically comes down to saying that certain singularities in 
the hyperelliptic family and singularities in the complementary family are conjugate under 
the global monodromy. 
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Lemma 3.8 Let £ in G hypeT denote the monodromy element which acts by the Dehn twist 
_i on the cohomology of the hyperelliptic piece. Let n G G 9 90 denote a Dehn twist 
coming from a double point on the complement of U. Then there is an element ip of the 
global monodromy group G such that ip' 1 ^ = n. 

Proof. This is because the Dehn twists generating the fundamental group of FH°(Z, C) — B> 
are all conjugate since the discriminant divisor IB) is connected. However, one must be a 
bit careful since we are looking at cohomology with coefficients in a local system that is 
not necessarily preserved by the full fundamental group: our representation p is preserved 
by a subgroup of finite index which corresponds to a covering of ¥H°(Z, C) ramified along 
D, and in this covering the inverse image of the discriminant locus might no longer be 
connected. We remedy this by stating somewhat more explicitly how to construct ip, but 
without actually writing down the equations for this loop in FH°(Z, C) since that would be 
tedious. Recall that we have assumed that the last two branch points of the hyperelliptic 
curve were in branch cuts where the passing transformation was the identity. The curve 
acquires a node when these two branch points come together; this node corresponds to the 
monodromy element £. Then this node can move out of our coordinate patch U and into the 
complementary region Z — U . At this point we are left with a hyperelliptic handle whose 
equation is a small deformation of x m ~ 2 — y 2 rather than x m — y 2 . The connectedness of 
the discriminant locus analogous to B<g — Goo but for m — 2 rather than m, and also for 
two nodes at once, allows us to choose a path whereby the node which came out of U gets 
interchanged with another node corresponding to n. This connectedness statement, which 
holds for k large enough, is the second statement referred to in condition (d) at the start 
of the argument. After interchanging the two singularities, go backwards along the path 
to send the £ node back into U. All of this corresponds to a path in the subvariety of D 
corresponding to sections with two nodes. (Geometrically this subvariety is the codimension 
two nodal locus of the discriminant variety; we choose a path which interchanges the two 
sheets of the discriminant which come together along the nodal locus.) Choose a path ip 
which is near to this path, but in the complement of the discriminant. This has the effect 
of giving the conjugation above. Finally, notice that all of this took place in a region where 
the representation p is trivial, so we can follow p along the path ip, i.e. the path ip lifts to a 
path in the ramified covering on which p is defined. □ 

Let £ G G hyper , i] G G 9 ~ 90 , and ip G G be the elements from the above lemma. Note that 
n is a generating symplectic transvection in Sp(2(g — go)). Of course n no longer acts as a 
symplectic transvection but as a direct sum of n 2 copies of a transvection on H 1 (X a , C) 9 ~ 90 . 
Now the proof of Theorem IB1 will be a consequence of the following statement. 

Lemma 3.9 The global monodromy group is the full product 

G = Sp(H\X ,C))x J] MH\X ,W U ), 



41 



acting on H 1 (X Q , C) © {® u ^(i,o)H l (X ,W u )) by the sum of the standard representations . 

Proof. Note first that G is contained in the product. Its image in the first factor is the full 
group Sp(2g) = Sp(if 1 (X , C)), which is just the usual statement (the Deligne-Kazhdan- 
Margulis theorem again) for cohomology with the trivial coefficient system C. 

Look at one of the pieces if 1 (X Q ,W u ), proceeding in the spirit of the hyperelliptic dis- 
cussion in the previous section (the proof of Lemma 13.21 and Theorem [XJ . First we show 
that the action of G on iJ 1 (X , W„) is irreducible. For this, it suffices to consider the action 
of the group algebra. There is a vector v in H 1 (X Q , W u ) hyper such that 

C[G hyper ] ■ v = H\X ,W u ) hyper . 

We claim that 

C[G]-v = H\X ,W u ). 

Let A be the image of the element £ — 1. It is a two-dimensional subspace of H 1 (X ,W U ), 
and in particular it is contained in C[G] - v. Thus there are elements V\ and v-i of C[G] -v such 
that (£ — l)v i and (£ — l)v2 span A. On the other hand, the image B of r\ — 1 = , 0~ 1 (£ — l)ip 
is an isomorphic image of A, a two-dimensional space contained in H 1 (X ,W U ) 9 ~ 90 . The 
isomorphism is given by 

ijj' 1 : A ^ B. 

The vectors 

V> _1 (£ - 1)V>(V>"S) and - 

span i?. In particular i? is contained in C[G] ■ v. 

Now we can write H 1 (X Q , W U ) 9_S0 = C 2(9 ^ o) © C 2(9 " 9o) and the image of the action of 
G contains the diagonal copy of Sp(2(g — g )) (this is the image of G 9 ~ 90 ). The subspace 
B is transverse to the above decomposition, in other words it contains one basis element in 
each piece. It follows that the translates of B by elements of G 9 ~ 90 span H 1 (X , W u ) 9 ~ 90 . 
Therefore 

H^X^Wu) 9 ' 90 cC[G]-v, 

so putting this together with the above we get that C[G] ■ v = i/ 1 (X , W u ), so the action of 
G on ff 1 (JY" ,W u ) is irreducible as claimed (to get this last deduction we use the standard 
fact that G is semisimple so its action decomposes as a direct sum of irreducible pieces). 

Next note that the image of G acting on H 1 (X ,W U ) is a simple group. This is because 
the group G fhyper acting on if 1 (X ,W u ) contains an element whose Jordan normal form 
has a single Jordan block of length one (see the argument of the previous section for the 
hyperelliptic case). As before this implies that the image is simple. 

Next we show that the image of G acting on H 1 (X ,W U ) is the full symplectic group 
Sj)(H l (X , W„)). This again is by the same argument as in the previous section, using the 
fact that the image of G acting on W u contains a copy of Sp(2(<? — g )), and noting that we 
can insure that g — g is large enough (by choosing k big) . 
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Finally, complete the proof of Lemma 13.91 by noting that inside G hyper we can find an 
element which acts with different eigenvalues on each of the different pieces H l (X , W u ), the 
same element as exhibited in the previous section. As then, this implies that 

G = Sp(2g)x J] Sp(H\X ,W u )). 
This completes the proof of the lemma. □ 

Proof of Theorem [BJ The dihedral Heisenberg representation p which we chose here 
corresponds to a smooth point in Mb(X q , n), fixed under a finite index subgroup of the 
monodromy group. The action of the monodromy group which fixes p on the tangent space 
T[ p ]MB{X ,n) at p is exactly the action on H 1 (X Q , ad(p)) we considered above. The Zariski 
closure G as described in Lemma f3. 91 acts with an open orbit. Either from the general con- 
sideration in Lemma or else just by inspection, this monodromy group has no characters. 
Therefore by Corollary 12 .41 we get that the monodromy action on Mb{X , n) is Zariski dense. 
□ 

4 Further remarks 

4.1 Topologically irreducible families 

The requirement that our families have relatively large monodromy was used in an essential 
way in the proofs of Theorems El and El However this requirement seems to be more an 
artifact of the method of proof rather than a real condition on the family / : X — > B which 
is necessary for the density of the monodromy action. In this section we briefly examine 
some consequences of the density, which will allow us to probe the necessity of the 'large 
monodromy' condition. 

Recall that a smooth family of curves / : X — > B is called topologically irreducible if and 
only if there is no finite collection of disjoint embedded circles in X a which is preserved by 
the geometric monodromy. We have the following simple 

Lemma 4.1 Let f : X —>■ B be a family of smooth curves, such that the monodromy action 
ofni(B,o) has a Zariski dense orbit on the Betti moduli space Mb(X ,u) for some n > 1. 
Then f : X — > B is topologically irreducible. 

Proof. Assume that one can find simple disjoint loops ai,...,at C X Q such that the 
collection {ai, . . . ,ak} of free homotopy classes on X a is preserved by mon(7ri(£>, o)) C 
Map(A ). Then for every JV e Z we have a well defined mon(iri(B, o))-invariant regular 
function ip N : M B (X a ,n) -> C on M B (X ,n), given by V'jv(M) := r ^ r (Ili=i(p( a i)) N ) ■ But 
clearly for some N the function ip^ will be non-constant and so mon(7Ti(S, o)) can not have 
a Zariski dense orbit on Mb(X ,ti). The lemma is proven. □ 
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In particular, all families of curves satisfying the hypothesis of Theorem |X] and El will 
be topologically irreducible. Recently C. McMullen has shown that topological irreducibil- 
ity holds very generally: every non-isotrivial holomorphic family of curves is topologically 
irreducible Mc MullenOCH Proof of Theorem 3.1]. In particular, this corollary of our main 
results was already known. 

When we started the current project we were hoping that topological irreducibility will 
allow one to distinguish symplectic Lefschetz pencils (whose topology tends to be much 
softer) from projective Lefschetz pencils. In the meantime however, Ivan Smith succeeded 
in showing [SmithOlJ that all symplectic Lefschetz fibrations over P 1 are topologically irre- 
ducible. We still expect that the stronger property GZD (or the open orbit property from 
Theorem [BJ) will allow one to distinguish projective from symplectic Lefschetz pencils. We 
hope to return to examples of this type in a future paper. 

McMullen's result (with the alternative symplectic proof by Smith when the base is P 1 ) 
is the only evidence we have for the following conjectural generalization of Theorems 1X1 and 

m 

Conjecture 4.2 For a family f : X — > B assume that mon(7r 1 (5, o)) is not a finite group. 
Then: 

(i) There is no meromorphic function on Ms(X ,?7,) an which is invariant under the action 

of mon^(7r 1 (5, o)) (equivalently there is no meromorphic function on M DR (X Q , n) an 
which is mon^ Jj ,(7Ti(S, o)) -invariant); 

(ii) In the case of Mb{X ^ti), considered with its natural structure of an ajfine algebraic 

variety, there exist a point x B £ M B (X ,n) so that the orbit 

mon^(7r 1 ( J B,o)) ■ x B C M B (X ,n) 
is Zariski dense in M B (X ,n). 

Procesi's theorem |Procesi74"j implies that the field of rational functions on M B (X Q ,n) is 
generated by traces of evaluation maps for conjugacy classes of simple loops on X . One 
might hope (although we didn't find an argument) that the field of ni(B, o)-invariant ratio- 
nal functions on M B (X a ,n) is similarly generated by the traces of evaluation maps at finite 
invariant collections of simple loops. If this were the case then McMullen's theorem would 
imply the validity of the variant of Conjecture 14.21 concerning algebraic meromorphic func- 
tions. The property AGZD1 (i.e. the conjecture as it is stated using analytic meromorphic 
functions) would seem to remain more elusive. 
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4.2 Points T-near to a finite representation 

We briefly describe here another variation on the basic result. Essentially, we have con- 
structed a finite-image representation, the dihedral Schrodinger representation p, which cor- 
responds to a smooth point in M#(X ,n) and which turns out to be sufficient in order to 
get the Zariski-denseness property. In an attempt to better understand what is going on, we 
can explore a bit further the sense in which p is near the rest of Mb(X , n). 

Suppose a finitely presented group T acts on an affine variety M, and suppose p G M is a 
closed point in the smooth set of M, fixed by the action. We say that another point q G M 
is T-near to p if p lies in the closure of the orbit T ■ q. Let iV = Near(M, p, T) C M denote 
the subset of points q which are T-near to p. It is T-invariant. Let T p N denote its tangent 
cone at p, defined to be the set of limits of secants to M going from p to points q G N which 
approach p (the limits of secants may be taken in any real embedding of M). Note that 
T p N C T p M is an invariant subset of the tangent space to M at p. An easy argument similar 
to that of Corollary 12.41 shows that if T P N is Zariski-dense in T P M then N is Zariski-dense 
in M. 

Suppose 7 G T. Let R C T P M denote the span of the eigenvectors of 7 whose eigenvalues 
have norm < 1. We claim that R C T P N. Choose a smooth submanifold V G M tangent to 
R. Let D be a ball neighborhood of p in M. Then the collection 



is a collection of manifolds with boundaries lying in the boundary of D, which are tangent 
to R at the origin, and whose curvature is bounded (the V H 7 fc -D all lie in a sector preserved 
by 7 _1 and in which 7 _1 smooths things out). Thus these converge to a manifold V°° which 
is preserved by the action of 7 and on which 7 acts with all eigenvalues < 1. In particular, 
V°° C N which shows that R C T p N. 

Lemma 4.3 Suppose p G M^(X ,n) is the dihedral Schrodinger representation we have 
considered above. Suppose that a group T = tti(B,o) acts, satisfying one of the hypotheses 
of Theorem A or Theorem B. Let N = Near(Ms(X , n), p, T). Then T P N is Zariski-dense 
in T p M B (X ,n). 

Proof. Recall that 



such that the monodromy group T is Zariski-dense in the product G = Y[ u G u with G u = 
Sp(i/ 1 (X , W u )). Each component of the decomposition corresponds to a weight one vari- 
ation of Hodge structure over B which is irreducible and not unitary (since both Hodge 



{V k :=-f- k (Wn-f k D)} 



T [p] M B (X ,n) = H\X , a d(p)) 



and that we have a decomposition 
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subspaces are nontrivial). Therefore T actually lies in a real form which decomposes 

r c Gr = Y\ c Uj k 

u 

and the G Ui r are noncompact real forms of the symplectic groups. (One cannot have a 
real component whose complexification splits into two components, because that would be 
a complex group considered as a real group, which is never of Hodge type.) The real Zariski 
closure of T, i.e. the intersection of all real algebraic subsets of G containing T, is Gr, since 
anything smaller would lead to a smaller complex Zariski closure. 

Let pr u : Gr — > G Ui r denote the projection. Let E u C Gr be the real algebraic subset of 
elements g such that all of the eigenvalues of pr u (g) G G Uj r have norm 1. This is a proper 
subset since G Uj r is noncompact. The union (J u E u is again a proper real algebraic subset of 
Gr, so it cannot contain Y. 

Thus there is an element 7 G V such that every projection pr u (7) has at least one 
eigenvalue of norm different from 1. On the other hand these projections have determinant 
one, so each pr u {^) has at least one eigenvalue of norm < 1. 

In particular, there is a vector v G if 1 (X , ad(p)) such that v is in the span of the 
eigenvectors of 7 corresponding to eigenvalues of absolute value < 1, and such that v has a 
nonzero component in all of the irreducible factors of H 1 (X Q , ad(p)). Using the fact that the 
complex Zariski closure of T contains a product of symplectic groups (corresponding to the 
decomposition into irreducible pieces of the representation H l (X 0) ad(p))), we find that the 
orbit of the vector v under the action of V is Zariski-dense in T[ p ]Mb(X , n). On the other 
hand, from our discussion prior to the present lemma, v is in the T-invariant subset T P N . 
Thus T p N is Zariski-dense. □ 

Corollary 4.4 The moduli space M#(X ,n) contains a smooth point, the dihedral 
Schrodinger representation p, such that the set of points Near(M#(X , n), p, T) C M#(X , n) 
which are T-near to p, is Zariski-dense in Mb(X ,7i). 

It is clear that any T-invariant regular (i.e. holomorphic algebraic) function takes the 
same value at p as at every point of Near(Mg(X , n), p, T). In particular, this corollary 
implies the result that there are no T-invariant regular functions. This result is weaker than 
our main results about nonexistence of invariant meromorphic functions, but does provide a 
slightly different conceptual route to the topological irreducibility result refered to above. 

Our motivation for introducing the notion of T-nearness is that the first and second 
authors asked some time ago whether there was any sense in which the finite-image repre- 
sentations could take up a big place in Mb{X ,u). The short answer to that question is 
that, by Jordan's theorem, the finite image representations occupy a rather small place in 
that there are only finitely many outside of representations which factor through a normal- 
izer of a torus (and those which factor in this way lie in a closed subset of relatively high 
codimension). However, Corollary 14.41 provides the slightly more subtle answer that, in the 
presence of a large monodromy action, if you start out very near to a certain finite-image 
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represenation (such as one of our dihedral Schrodinger representations) and then let the 
monodromy act, then you can get out to a significant part of M B (X ,n). 

4.3 Other groups 

Finally, let us explicitly state that we expect that all of the results and conjectures of this 
paper to hold for Betti and deRham cohomology with coefficients in an arbitrary complex 
reductive group G. Specifically we make the following 

Conjecture 4.5 Let f : X — > B be a smooth algebraic family of curves and G a complex 
reductive group. Then: 

(i) Assume that f is not isotrivial. Then the families 

M B (X/B, G)^B and M DR (X/B, G) -> B 

of relative Betti and de Rham cohomology with coefficients in G are GZD when 
equipped with the non-abelian Gauss-Manin connection. 

(ii) Assume that f comes from a projective Lefschetz pencil of sufficiently high degree. Then 

there exists a smooth point p G M B (X ,G) which is fixed by a finite index subgroup 
T C iri(B,o) and for which the Zariski closure ofT in GL(T p M B (X , G)) acts with an 
open orbit on T p M B (X Q ,G). 
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